S cikit-talk: A toolkit for processing real-world conversational speech data (2021)

Abstract: We present Scikit-talk, an open-source toolkit for processing collections of real-world conversational speech in Python. First of its kind, the toolkit equips those interested in studying or modeling conversations with an easy-to-use interface to build and explore large collections of transcriptions and annotations of talk-in-interaction. Designed for applications in speech processing and Conversational AI, Scikit-talk provides tools to custom-build datasets for tasks such as intent prototyping, dialog flow testing, and conversation design. Its preprocessor module comes with several pre-built interfaces for common transcription formats, which aim to make working across multiple data sources more accessible. The explorer module provides a collection of tools to explore and analyse this data type via string matching and unsupervised machine learning techniques. Scikit-talk serves as a platform to collect and connect different transcription formats and representations of talk, enabling the user to quickly build multilingual datasets of varying detail and granularity. Thus, the toolkit aims to make working with authentic conversational speech data in Python more accessible and to provide the user with comprehensive options to work with representations of talk in appropriate detail for any downstream task. For the latest updates and information on currently supported languages and language resources, please refer to: https://pypi.org/project/scikit-talk/

## introduction real-world conversational speech data, also known in the designated fields of study as transcription of talk-in-interaction, is complex. talk can be transcribed in varying level of detail and symbolic representations of elements in talk range from simple word-level transcripts to fine-grained phonetic representations and multi-layered xml tags for all sorts of audio-visual information that the representation aims to capture. when using this data type for any downstream task, an important step is to decide on the appropriate level of granularity and to strive for a systematic representation format. scikit-talk is designed to aid anyone interested in processing talk with these decisions and to make building datasets of talk more efficient, flexible and accessible. designed for applications in speech processing and conversational ai, scikit-talk provides tools to custom-build datasets for tasks such as intent prototyping, dialog flow testing, and conversation design. an important step in the development of commercial task-oriented bots is the initial design of intent labels and dialog flows. depending on the task, prototyping intent flows can be difficult. breaking down the interaction of ordering a pizza into discrete intents such as "select pizza type" and "input delivery address" is straight-forward. but, for instance, designing dialog flows for more complex interactions, such as bots for technical support tasks, help, or faq hotlines can be more complex. how will these interactions unfold? what are typical dialog flows for such interactions in the real world? scikit-talk provides an accessible tool for tasks such as bot prototyping for conversational ai. it offers a unified toolkit to process and query databases of real-world interactions to find and analyse instances of a certain type of interaction for data-driven conversation design and development cycles (figure 1 ). prototype interaction flows sometimes do not match well with how users actually complete the task in real-world scenarios, which can result in additional development cycles until a satisfactory performance is reached @xcite . initial design decisions may require revision later on, such as editing, merging or deprecating intents once real users interact with the bot. scikit-talk addresses this by providing builders and designers with a tool to explore how to better match intent labels and dialog flows in the prototype stage. for instance, when prototyping of a bot for a customer support task, scikit-talk can be used to explore similar explanatory sequences in real-world data. this helps builders to ground their prototype in authentic data and mitigate the risk of grounding development cycles on artificial premises. scikit-talk is the first open-source tool for processing transcripts of authentic speech specifically designed for research and development. it comes with a range of data exploration tools for collocation and concordance analysis, as well as a pipeline to automate annotation, analysis, and visualization of interaction types or specific nlu intents (such as greeting, accusing, deflecting, or apologizing). in contrast to prototyping based on typed chat or movie subtitle data, scikit-talk enables insights that take more detail of real talk into account, including seemingly incoherent or messy discourse structures, disfluency, repair, repetition or restarts and other features that are common in natural spoken interaction. designing for real talk means designing closer to real-world scenarios. figure 2 shows an example of intent label and dialogue state design grounded in conversational data. the analysis highlights some of the issues that arise when working with this data type, such as labelling various turns and dealing with features of conversational interaction such as abandoned turns or joint turn completions. inspired by scikit-learn, the toolkit consists of several modules to preprocess, explore and analyse this data type that were designed with explainability and customisation in mind @xcite . the toolkit also enables the user to combine existing language resources using pre-built interfaces for common corpus transcription formats. the motivation behind scikit-talk is to build a tool to conveniently explore large collections of authentic talk. this aim is reflected in the following design decisions: the idea of a toolkit for conversational speech processing is inspired by an existing similar tool for online chat conversations @xcite . scikittalk aims to provide both a unified framework for several existing transcription conventions as well as a range of tools to explore this data type in python. the preprocessor module is used to combine several existing datasets via pre-built interfaces for various data formats while preserving as much transcribed information as possible. apart from corpus building, scikit-talk also includes tools for data manipulations. it includes string matching tools to explore and identify features of interest. and it includes tools for unsupervised machine learning techniques to explore distributional patterns in the data. as for custom datasets, scikit-talk can be used to analyse any transcriptions of talk in a turnbased format where the conversation unfolds in the "abab..." pattern for two speakers or "abcbca..." for three speakers, etc. foot_0 ## pre-built corpus interfaces the main challenge of providing a comprehensive tool to explore how people talk in natural settings distributionally is that language resources of natural conversational speech are few and far between. unlike movie subtitle corpora, these datasets are small because building them requires painstaking manual transcription and annotation. existing large-scale conversational corpora often come with a custom annotation formats. while this may be for good linguistic reasons, it poses a significant barrier to facilitate work across different datasets. processing and finding instances of a specific type of interaction or task in these datasets is hard because transcription conventions and data representation formats vary. addressing this challenge, scikit-talk provides a platform that aims to collect and connect existing computer-readable transcription formats of talk. this fragmentation of representation formats poses a challenge to create custom datasets from multiple data sources or across existing corpora. scikit-talk here provides a practical solution by providing a module that converts data across several common formats, the childes chat format foot_1 , that of the spoken british national corpus (spo-kenbnc) foot_2 , and the upenn linguistic data consortium (ldc) 4 . this means the current version of the toolkit already covers several of the largest available datasets of authentic conversations, such as spokenbnc or any ldc datasets. more corpus interfaces for major language resources in more languages are planned. preprocessor enables the user to interface data from different corpora which can significantly reduce the workload of building large datasets and unifying data formats. the challenge here is that transcription formats may differ in convention and granularity, some providing more detailed information than others. preprocessor merges transcription features that are consistent across the formats while automatically discarding those that are not. this minimizes the loss of information for the sake of consistency. typically, consistency issues arise regarding the transcription of non-lexical conduct, pauses, overlaps, restarts and phonetic reductions. the module has been tested with spokenbnc, ldc and chat format data in british english and ldc and chat format data in mandarin chinese. combining several datasets using preprocessor results in a single, large dataset that enables the user to build collections of interactions that are more comprehensive, provide broader coverage of a phenomenon, and enable more methods of statistical analysis (figure 3 ). ## conclusion and future work scikit-talk provides a platform to represent spoken conversational data, build datasets, and explore how certain types of interaction unfold in authentic talk. tailored to the research and development community, the open-source toolkit makes working with transcriptions of talk across corpora more accessible for anyone interested in processing this data type. scikit-talk features a preprocessor tool that provides pre-built corpus interfaces for (currently three) popular transcription formats and enables rapid construction of unified data representations. the explorer, annotator, and cluster analysis modules provide streamlined data exploration tools for collections of specific interaction or intent types using various string matching and unsupervised machine learning techniques. in the future, scikit-talk will be extended to provide additional corpus interfaces, covering more resources in more languages. our aim is to maintain compatibility with future major releases of language resources of real-world conversational speech and to provide the research community with a useful tool for conversational speech processing to study and model talk in all its glori-