Improving Segmentation for Technical Support Problems (2020)

Abstract: Technical support problems are often long and complex. They typically contain user descriptions of the problem, the setup, and steps for attempted resolution. Often they also contain various non-natural language text elements like outputs of commands, snippets of code, error messages or stack traces. These elements contain potentially crucial information for problem resolution. However, they cannot be correctly parsed by tools designed for natural language. In this paper, we address the problem of segmentation for technical support questions. We formulate the problem as a sequence labelling task, and study the performance of state of the art approaches. We compare this against an intuitive contextual sentence-level classification baseline, and a state of the art supervised text-segmentation approach. We also introduce a novel component of combining contextual embeddings from multiple language models pre-trained on different data sources, which achieves a marked improvement over using embeddings from a single pre-trained language model. Finally, we also demonstrate the usefulness of such segmentation with improvements on the downstream task of answer retrieval.

## introduction problems, reported by users of software or hardware products -called tickets or cases, are often long and complex. along with a description of the problem, users often report the setup, steps they have tried at mitigating the problem, and explicit requests. these problems also contain various nonnatural language elements like snippets of code or commands tried, outputs of commands or software tools, error messages or stack traces, contents of log files or configuration files, and lists of key-value * work done at ibm research during a summer internship † now at google figure 1 : various non-natural language segments labelled from a problem on askubuntu pairs. figure 1 shows a sample support problem from askubuntu foot_0 , where all such segments are labeled. while these segments are important sources of information for the human reader, they are difficult to handle for systems built to automatically answer support problems. as noted in @xcite , the non-natural language segments lead to parsing mistakes, and errors in the understanding of support problems. correctly identifying these segments can also augment problem understanding. for instance, a retrieval engine with error messages and their solutions indexed in distinct fields would return better results with a fielded query containing just the error message from the ticket. specialized tools for log analysis @xcite could also be run specifically on the identified log segment of problems. in this paper, we aim to address the problem of identifying and extracting these non-natural language segments from support tickets. in particular, we choose to focus on the following six segment labels which appear often in support tickets (also shown in figure 1 ): we formulate the problem as a sequence labelling task, with word-level tags used to encode segments. to leverage the rich literature of supervised approaches in this framework, we also create a dataset with segments tagged for questions from askubuntu foot_1 . our contributions are as follows -1. we introduce a novel task towards understanding technical support problems, which has implications on a variety of downstream applications. we also release a tagged dataset of problems for the task. this hopefully provides direction for future research. our model leverages activations from multiple language models pre-trained on different data sources, and we show how they can be used to improve performance on the task. ## related work understanding technical support problems is a particularly difficult task, owing to the long text of problems. in @xcite , the authors propose that understanding can be approached by extracting attributes of the ticket that correspond to the description of the problem (symptom), steps taken for mitigation (attempt), and explicit requests (intent). they also propose a dependency parser-based approach for extracting these attributes. however, while this approach pays attention to the semantics of the problem, the syntactical idiosyncrasies are ignored. the idea of segmenting of questions for improvements on downstream tasks is not new. in @xcite , the authors propose an unsupervised graph-based approach for segmenting questions from community question answering (cqa) websites into sub-questions and their related context sentences. the authors demonstrate improvements in question retrieval by using these segments for more granular similarity matching. @xcite uses representations from a character-level language model for segmenting code spans in stack overflow posts. the author uses code tags in html sources of posts for supervised training of a character level sequence labelling model. however, the code tags in the posts usually include all forms of non-natural language text like code snippets, command outputs, error messages or stack traces, and file paths (see fig 2 ). the resulting level of granularity is thus insufficient for effective application in downstream tasks such as automated problem resolution. the task of textsegmentation in itself has been well studied in the literature, with popular unsupervised approaches like texttiling @xcite and c99 @xcite . while, the problem of ticket segmentation, as defined by us, involves both segmenting and identifying segment types, we compare the performance of a more recent supervised segmentation approach @xcite against our proposed model. significant amount of work has been done on using sequence labelling approaches for text segmentation tasks (huang et al., 2015; chiu and nichols, 2016; lample et al., 2016; ma and hovy, 2016; rei et al., 2016; peters et al., 2017). in wang et al. (2018) the authors use elmo embeddings and a bilstm-crf based architecture with selfattention for the task of neural discourse segmentation. we adopt a similar architecture, and explore the effect of using pre-trained contextual embeddings on our task. given the fact that different segments in technical support problems have very different vocabularies, we also explore leveraging pre-trained language models on a variety of different datasets. ## data our dataset is derived from english questions on ask ubuntu. questions posted on the website are similar to proprietary tech support tickets (in terms of question length, number of keywords/noun phrases, etc). we would like to point out that while posts on the website support the code html tag, it is not granular enough for our downstream tasks. these tags are also often abused to present snippets of command outputs/error messages/file paths etc. figure 2 shows examples of such questions. we also do not use other metadata available (like turnbased information) with the data dump because these are not available with proprietary tickets. tagging is performed at the word level, and we use the bio tagging scheme. we have a pair of begin and inside tags for each of the 6 non-natural language segments, and natural language segments are labelled o, totalling to 13 tags. we use the doccano tool foot_2 for labelling, which provides better support for labelling long chunks in big documents compared to other popular sequence labelling annotation tools. we obtain labelling for 1,317 questions, totalling to 11,580 spans (including spans labelled as o) and over 1.18 million words. we divide the data into 80:10:10 train, val, and test splits, at random. high-level statistics for the dataset are presented in in figure 4 we show the inter-annotator agreement between two annotators on 50 questions. few of the label pairs with large off-diagonal values include - the low inter-annotator agreement (κ = 0.7637) illustrates the inherent difficulty of the task. at this point, it's important to note that while there's some confusion in identifying labels for these segments, the need for these separate labels stems from downstream tasks. ## model given a technical support question, we formulate the segmentation problem as a sequence labelling task. it is an intuitive choice, given its efficacy for similar text segmentation problems like discourse segmentation @xcite and chunking @xcite . figure 5 presents an overview of our model. we explore different embeddings for each word (character-based embeddings, pretrained embeddings, and pre-trained contextual embeddings). these word embeddings are then fed to a bi-directional gru for encoding context. on the output of the gru layer, we explore the effect of attention. finally, the representations are passed to a crf to decode the segment labels. we also study the impact of combining pre-trained contextual embeddings from multiple language models, trained on different data sources. in the rest of this section we detail individual components of the model. ## experimental setup with the setup above, we study the performance of various model components on the task of segmenting support problems. to put the performance in perspective, we also compare against three baselines detailed in section 5.1. the evaluation metrics are carefully selected, avoiding an exact evaluation of such long and noisy segments, and rewarding partial retrieval of segments. the chosen evaluation metric is discussed in section 5.2. finally, to demonstrate the usefulness of the task, we evaluate the performance of answer retrieval with segmentation (section 5.3). all baselines and sequence labelling models are trained on the train split, and fine-tuned on the validation split. for the baselines, we only tune the regularization strength parameter. for the sequence labelling model, we tune the dropout and recurrent dropout parameters, as well as the learning rate. our best performing models have a dropout of 0.3, recurrent dropout of 0, and learning rate of 1e-3. all results are then reported on the test split. ## results as one would expect, the context baseline performs much better than the sentence only baseline. the sequence labelling models, however, outperform both the baselines by a huge margin, demonstrating the effectiveness of the model on the task. specifically, the best performance is achieved by combining pre-trained embeddings from multiple language models trained on different data sources. it significantly outperforms the model using embeddings from a single pre-trained model on english (explored in section 6.3). in the following section we present results from the various model components we explored. ## conclusion in this paper, we introduce and address an important problem towards a better understanding of support tickets -segmentation of various nonnatural language segments. we create an annotated dataset for the task, on questions from the publicly available website, ask ubuntu. we also study the performance of the most recent recurrent neural network-based approaches to sequence labelling, on this task. in the end, we propose the novel idea of combining pre-trained embeddings from language models trained on different data sources, which substantially improves performance. we also demonstrate the usefulness of the task with improvements in retrieval of the correct answer. our future research direction includes a thorough study of differences in this dataset with actual tickets, and potential for transfer. it is still valuable to study models on open datasets, however, as these are readily available to the community.