C onvo K it: A Toolkit for the Analysis of Conversations (2020)

Abstract: This paper describes the design and functionality of ConvoKit, an open-source toolkit for analyzing conversations and the social interactions embedded within. ConvoKit provides an unified framework for representing and manipulating conversational data, as well as a large and diverse collection of conversational datasets. By providing an intuitive interface for exploring and interacting with conversational data, this toolkit lowers the technical barriers for the broad adoption of computational methods for conversational analysis.

## introduction the nlp community has benefited greatly from the public availability of standard toolkits, such as nltk @xcite , stanfordnlp @xcite , spacy (honnibal and montani, 2020), or scikit-learn @xcite . these toolkits allow researchers to focus on developing new methods rather than on re-implementing existing ones, and encourage reproducibility. furthermore, by lowering the technical entry level, they facilitate the export of nlp techniques to other fields. although much of natural language is produced in the context of conversations, none of the existing public nlp toolkits are specifically targeted at the analysis of conversational data. in this paper, we introduce convokit ( https://convokit.cornell. edu ), a python package that provides a unified open-source framework for computationally analyzing conversations and the social interactions taking place within, as well as a large collection of conversational data in a compatible format. in designing a toolkit for analyzing conversations, we start from some basic guiding principles. firstly, conversations are more than mere 'bags of utterances', so we must capture what connects utterances into meaningful interactions. this trans-lates into native support of reply and tree structure as well as other dependencies across utterances. secondly, conversations are inherently social. people often engage in multiple conversations, and how we understand interactions is contingent on what we know about the respective interlocutors. similarly, the way we understand each speaker is contingent on their entire conversational history. thus, a conversational analysis toolkit must allow for the integration of speaker information and behaviors across different conversations. thirdly, conversations occur in vastly different contexts, from dyadic face-to-face interactions, to discussions and debates in institutional settings, to online group discussions, and to large-scale threaded discussions on social media. this means that the toolkit must offer a level of abstraction that supports different interaction formats. finally, since conversational data is key to many social science fields (e.g. political science, sociology, social psychology), the framework should be accessible to a broad audience: not only experienced nlp researchers, but anyone with questions about conversations who may not necessarily have a high degree of nlp expertise. in this paper, we describe how these principles guided our design of convokit's framework architecture (section 2), describe some of the analysis methods (section 3) and datasets (section 4) included in convokit, and conclude with some highlevel remarks on future developments (section 5). ## framework architecture the current state of the software and data ecosystem for conversational research is fragmented: popular conversational datasets are each distributed in different data formats, each using their own task-specific schemas, while similarly, code for reproducing various conversational methods ## transformers in this section, we introduce some of the built-in transformers that are available for general use. broadly speaking, we can group the functionality of transformers into three categories: preprocessing, feature extraction, and analysis. preprocessing refers to the preliminary processing of the corpus objects prior to some substantive analysis. for example, at the utterancelevel, preprocessing steps can include converting dirty web text into a cleaned ascii representation (implemented in textcleaner) or running a dependency parse (implemented in textparser). at the conversation-level, preprocessing steps might include merging consecutive utterances by the same speaker, while at the speaker-level, they feature extraction refers to transformation of conversational data, such as utterance text or conversational structure, into (numerical) features for further analysis and applications. an example of an utterance-level feature extractor is the previously described politenessstrategies, while an example of a conversation-level feature extractor is hyperconvo, which constructs a hypergraph representation of the conversation and extracts features such as (generalized) reciprocity, indegree and outdegree distributions, etc. analysis is the process of combining utterance, conversation and speaker features and metadata into a statistical or machine learning model to achieve a higher-level understanding of the corpus. for example, fightingwords implements @xcite 's method for prinicpled comparison of language used by two subsets of a corpus; classifier acts as a wrapper around any scikit-learn machine learning model and can be used to classify utterances, conversations, or speakers based on the output of feature extraction transformers; and forecaster implements chang and danescu-niculescu-mizil (2019)'s method for modeling the future trajectory of a conversation. figure 2 illustrates how transformers belonging to each category can be combined in sequence to perform a practical conversational task: comparing the language used in movie dialogs containing characters of different genders to that used in dialogs containing only one gender. foot_0 4 datasets convokit ships with a diverse collection of datasets already formatted as corpus objects and ready for use 'out-of-the-box'. these datasets cover the wide range of settings conversational data can come from, including face-to-face institutional interactions (e.g., supreme court transcripts), collaborative online conversations (e.g., wikipedia talk pages), threaded social media discussions (e.g., a full dump of reddit), and even fictional exchanges (e.g., movie dialogs). 2 the diversity of these datasets further demonstrates the expressiveness of our choice of conversation representation. we also provide guidelines and code for transforming other datasets into con-vokit format, allowing convokit's reach to extend beyond what data is already offered. ## conclusions and future work in this paper, we presented convokit, a toolkit that aims to make analysis of conversations accessible to a broad audience. it achieves this by providing intuitive and user friendly abstractions for both representation and manipulation of conversational data, thus promoting reproducibility and adoption. convokit is actively being developed. while it is currently heavily centered around text analysis (with other modalities being only indirectly supported as metadata), providing first-class support for spoken dialogs is considered as an important line for future extension. in addition, we aim to continue to incorporate new datasets, analysis methods, and integrate with other parts of the nlp software ecosystem that could benefit from con-vokit's abstractions, including dialog generation toolkits such as parlai @xcite . convokit is an open-source project and we welcome contributions of any kind, ranging from bugfixes and documentation, to augmenting existing corpora with additional useful metadata, to entirely new datasets and analysis methods. 3