Automatic Topic Labeling in Asynchronous Conversations

Asynchronous conversations are conversations where participants collaborate with each other at different times (e.g., email, blog, forum). The huge amount of textual data generated everyday in these conversations calls for automated methods of conversational text analysis. Topic segmentation and labeling is often considered a prerequisite for higher-level conversation analysis and has been shown to be useful in many NLP applications including summarization, information extraction and conversation visualization. Topic segmentation in asynchronous conversation refers to the task of clustering the sentences into a set of coherent topical clusters. In [6, 7], we presented unsupervised and supervised topic segmentation models for asynchronous conversations, which to our knowledge are the state-of-the-art. This study concerns topic labeling, that is, given a set of topical clusters in a conversation, the task is to assign appropriate topic labels to the clusters. For example, five topic labels in a Slashdot [2] blog conversation about releasing a new game called Daggerfall are game contents and size, game design, bugs/faults, other gaming options and performance/speed issues. Such topic labels can serve as a concise summary of the conversation and can also be used for indexing. Ideally, topic labels should be meaningful, semantically similar to the underlying topic, general (i.e., broad coverage of the topic) and discriminative (or exclusive) when there are multiple topics [10]. Traditionally, the top K terms in a multinomial topic model (e.g., LDA [3]) are used to represent a topic. However, as pointed out by [10], at the word-level, topic labels may become too general that it can impose cognitive difficulties on a user to interpret the meaning of the topic by associating the words together. On the other hand, if the labels are expressed at the sentence-level, they may become too specific to cover the whole theme of the topic. Based on these observations, [10] and other recent studies (e.g., [9]) advocate for phrase-level topic labels. This is also consistent with the monologue corpora built as part of the Topic Detection and Tracking (TDT) project [1] as well as with our own email and blog conversational corpora in which human annotators without specific instructions spontaneously generated labels at the phrase-level. Considering all these factors, we also treat phrase-level as the right level of granularity for a label in this work. Few prior studies have addressed the topic labeling problem in different settings [10, 9]. Common to their approaches is that they first mine topics in the form of topic-word distributions from the whole corpus using topic models like LDA. Then, they try to label the topics (i.e., topic-word distributions) with an appropriate label using the statistical association metrics (e.g., point-wise mutual information, t-test) computed from either the source corpus or an external knowledge base (e.g., Wikipedia). In contrast, our task is to label the topical clusters in a given conversation, where topics are closely related and distributional variations are subtle (e.g., game contents and size, game design). Therefore, corpus-based statistical association metrics are not reliable in our case. Also at the conversation-level, the topics are too specific to find their labels in an external source. To our knowledge, none has studied this problem before. Therefore, there is no standard corpus and no agreed-upon evaluation metrics available. Our contributions aim to remedy these problems. First, we present a blog and an email corpora annotated with topics. Second, we propose to generate topic labels using an extractive approach, that finds the most representative phrases from the text without relying on an external source. Since, graph-based key phrase ranking has proved to be the state-of-the-art (unsupervised) method [11], we adopt the same framework. We propose a novel biased random walk model that exploits the fact that the leading sentences in a topic often carry the most informative clues for its label. However, the phrases extracted only by considering the sentences in a topic may ignore the global aspect of the conversation. As another contribution, we propose to re-rank the phrases extracted from the whole conversation with respect to the individual topics and include the relevant ones. Experimental results show that our approach outperforms other ranking models including a general random walk model (i.e., TextRank) proposed by [11], a leadoriented model and a frequency-based model, and including the relevant conversation-level phrases improves the performance.