Situation-Based Multiparticipant Chat Summarization: a Concept, an Exploration-Annotation Tool and an Example Collection (2021)

Abstract: Currently, text chatting is one of the primary means of communication. However, modern text chat still in general does not offer any navigation or even full-featured search, although the high volumes of messages demand it. In order to mitigate these inconveniences, we formulate the problem of situation-based summarization and propose a special data annotation tool intended for developing training and gold-standard data. A situation is a subset of messages revolving around a single event in both temporal and contextual senses: e.g, a group of friends arranging a meeting in chat, agreeing on date, time, and place. Situations can be extracted via information retrieval, natural language processing, and machine learning techniques. Since the task is novel, neither training nor gold-standard datasets for it have been created yet. In this paper, we present the formulation of the situation-based summarization problem. Next, we describe Chat Corpora Annotator (CCA): the first annotation system designed specifically for exploring and annotating chat log data. We also introduce a custom query language for semi-automatic situation extraction. Finally, we present the first gold-standard dataset for situation-based summarization. The software source code and the dataset are publicly available.

## introduction in the recent years, the attitude to multiparticipant chat has changed: what was regarded as a distraction is now used as primary means of communication in both professional and personal environments. however, its evident problems, such as the inability to quickly and efficiently navigate a large body of skipped messages, are yet to be addressed. one of the ways of addressing this is summarization. however, due to the specifics of text chat data, such as noise and length, no widely accepted model for this task has been created yet. nevertheless, there have been notable works in the field. one of them is collabot @xcite : a fully-fledged chat summarizer, which, however, never went public. additionally, there is a considerable body of work on email summarization, such as @xcite , @xcite , @xcite , which present both annotated data and a summarization approach. while these works are an indispensable basis for the research in the area, we believe that chat data possesses enough specific qualities (such as extremely short message length, presence of specific slang and emoticons, and largely informal grammar and spelling) to warrant new annotation procedures and summarization methods. to the best of our knowledge, publicly available annotated data for this task is both rare and, additionally, highly specific. most of the aforementioned works have created their own specific annotation procedures and applied them to small volumes of data. annotated data is hard to obtain in and of itself, and creating a gold-standard dataset from noisy raw data may take a lot of effort and time. therefore, we have focused on creating a full-fledged annotation system for chat data. in the current paper, we propose novel annotation guidelines for multiparticipant chat data. in our vision, it would be most practical to summarize such datasets by specific situations. we define a situation as a subset of messages revolving around a single event in both temporal and contextual senses. the set of situation tags would be specific for each particular dataset, and devising a standardized tagset currently does not seem possible. each tagset would be devised by a human analyst and will be specifically suited for the needs of each user. this approach takes its roots in the ideas of open-domain event extraction, such as in @xcite , but differs from them on several points. first, we are interested in groups of documents. second, we do not explicitly extract event keywords. instead, we offer the user to decide what situations revolve around which events and how they are represented in the data. furthermore, the quality of the training data has to be very high. in our understanding, creating a gold-standard dataset requires full attention of a human annotator, and relying on automatic recommenders would yield inferior results. nevertheless, a recommender could be helpful for deep dataset exploration and for annotation assistance. such assistance may come in a form of generating candidates for manual cross-checks when the annotator had finished their job or for rapid dataset prototyping. since our task formulation is novel, there is no specifically trained machine learning (ml) model for it yet. to address the need for a recommender, we have designed a lightweight query language for rule-based detection of situations in chat datasets. next, we introduce chat corpora annotator, a standalone desktop application for exploring and annotating multiparticipant chat datasets. to the best of our knowledge, this is the first tool that addresses both these tasks simultaneously. additionally, we describe the annotation guidelines and the workflow for the summarization task. finally, we present an example collection that can be used to train machine learning models or serve as a gold-standard to assess summarization algorithms. the main contributions of the paper are: ## situations our inspiration for the proposed approach is based on cases such as a user taking a break from an important multi-participant chat for a significant amount of time. for example, it could be an employee taking a vacation. having returned, they would have to catch up with the rest of their colleagues, which would include browsing chat discussions that happened during their absence. therefore, they would be forced to navigate a large body of skipped messages which may be distracting and unproductive, as well as require a lot of time. basically, they would have to quickly look through all of the messages that were sent while they were away, since they would have no means to "prune" irrelevant discussions. the main issue here is the fact that they would not know whether a particular subset of messages is useful until they read at least some of them. another frequent scenario is a user searching for a particular conversation that is hard to find. usually, in this case user issues search queries trying different keywords. in general, chats offer unsophisticated search capabilities, limiting them to simplified textual search, thus hindering efficient retrieval. for example, if the user needs to recall the details of a meeting (for example, the name of the place their friends have agreed to go out to), they would to issue "bar", "pub", "restaurant" until they obtain the desired result. we propose to address such use-cases with chat log summarization that is based on the concept of situation. we define a situation as a subset of messages revolving around a single event in both temporal and contextual senses. we can propose many various examples: participants are arranging a meeting, selecting a product to be used in their project, solving a code issue and so on. an example of a situation that has been found and tagged with the use of our tool is presented in fig. 1 . in this figure, the user is shown a situation in which chat participants find a job offer and discuss the process of applying to it. our final goal that exceeds the scope of this paper is to build a system that would automatically detect such situations and present them to the user. the idea is to integrate the hypothetical tool into the interface of multiparticipant chat applications to provide the user with the means to take a situation-based perspective on chat history, instead of plain-text browsing as it has to be done currently. in its first version, we plan to highlight situation locations in the history, and then, in the future, present a generated summary. automatic extraction of situations can be performed using information retrieval, natural language processing @xcite , and machine learning (carenini and murray, 2012) techniques. however, currently, there are no corpora to train models for this problem and no gold-standard datasets to run experiments on. therefore, our first step is to create such a corpus and for this a special tool that assists tagging is needed. ## chat corpora annotator: system overview following the aforementioned considerations, we have created the chat corpora annotator (cca)the first exploration-annotation tool designed specifically for multiparticipant chat log data. its main use case is creating a dataset for the proposed summarization task. furthermore, the provided functionality can help gain clear and immediate insights into raw data. cca implements all common statistics and exploration tools and does not require any coding skills to use them. additionally, cca's csv viewer is more comfortable to use than, for example, the representation of a dataset that can be created with pandas foot_2 in a jupyter notebook. the user can resize and swap columns in the window without affecting the data. cca can be used for any textual data that contains a date, a username, and a text field, for example, email threads, twitter logs, etc. finally, all performance-heavy functionality is implemented separately from the main module and simply searching through a dataset does not require the user to load the corenlp models. ## evaluation we have conducted two kinds of evaluation tests: a responsiveness study and a usability study. ## related work in this section, we will review two types of related studies: annotation tools and corpora created from chat log data for various tasks. ## conclusion & future work in this paper, we have presented a novel situationbased summarization task, cca -an annotationexploration tool for large chat logs, a workflow for creating a situation-based summarization dataset, and an example corpus. chat corpora annotator offers a novel approach to exploration: a query language that allows the user to query a dataset for subsets of messages which could be a situation. to the best of our knowledge, cca is the only tool designed for these two tasks at once. further work on the tool will be focused on improving its usability and efficiency, as well as extending language support. the work on the summarization task will be moving towards implementing the first versions of the summarizer itself.