A multi-level approach for hierarchical Ticket Classification (2022)

Abstract: The automatic categorization of support tickets is a fundamental tool for modern businesses. Such requests are most commonly composed of concise textual descriptions that are noisy and filled with technical jargon. In this paper, we test the effectiveness of pre-trained LMs for the classification of issues related to software bugs. First, we test several strategies to produce single, ticket-wise representations starting from their BERT-generated word embeddings. Then, we showcase a simple yet effective way to build a multi-level classifier for the categorization of documents with two hierarchically dependent labels. We experiment on a public bugs dataset and compare our results with standard BERT-based and traditional SVM classifiers. Our findings suggest that both embedding strategies and hierarchical label dependencies considerably impact classification accuracy.

## introduction support tickets and incident reports are a valuable point of contact between customers and service providers @xcite . they are fundamental tools in the management of the relationship between businesses and users, allowing for the swift resolution of issues, thus leading to improved customer satisfaction, productivity, and compliance which service-level agreements (slas) @xcite . tickets can be derived from multiple communication channels, most commonly emails, specialized web forms, phone calls, live chats, and social media platforms @xcite . help requests are therefore logged as text, which represents the most important source of information to be used for automatic ticket management. being conversational by nature, tickets describe the issue or request in an often noisy and concise format @xcite . as a response to the increasingly high volume of these requests by customers, researchers have proposed the automation of various steps of the ticket resolution pipeline @xcite . these include the classification of tickets into broad topic categories (ticket classification) @xcite , the direct assignment of the issue to an expert capable of resolving it (expert finding) @xcite , as well as the direct resolution of tickets in a completely autonomous way (ticket resolution) @xcite . among these tasks, the accurate classification of incoming tickets within a pre-defined hierarchy of labels is among the most prevalent, as well as one of particular importance to ensure that these requests are dealt with swiftly. indeed, it is common for support tickets to be framed within a multi-level hierarchy such as the one just mentioned: each level of the hierarchy describes the issue at different levels of specificity. contributions this work will explore ticket classification (tic), a sub-task of text classification (tc), with the following objectives: our experiments show that both document summarization strategies and hierarchical information injection can contribute in a major way to tic accuracy. the code and datasets utilized in this work's experiments are made publicly available online foot_0 . ## related work the task of topical tic has been explored in recent works, which we briefly outline. relatedly, we also discuss recent advancements in the broader tc environment, as well as a short mention of dedicated hierarchical tc methods. ## the linux bugs dataset to evaluate a tic scenario, we experiment on a dataset of bugs crawled from the publicly available linux kernel bug-tracker foot_1 , as inspired by @xcite . the resulting linux bugs dataset contains tickets organized through the hierarchical dependent labels of "product" (e.g., network, drivers, etc.) and "component" (e.g., bios, scheduler, etc.). therefore, we utilize the former as main labels and the latter as sub-labels. to be precise, to avoid redundancies, we utilize the flattened labels as sub-labels, such as to differentiate sub-labels that share their name across main categories (e.g., network_other vs drivers_other). moreover, to reduce class imbalance, we discard all labels and sub-labels that appear less than 100 times. more details on the dataset, including an exemplary subset of the resulting hierarchy, reports on the labels' frequency and an example of the content of a ticket can be found in appendix b. ## methods the aforementioned bert lm has been one of the most popular contextualized lms since its inception. conceptually, bert is a bidirectional transformer-based neural network, made by stacking multiple encoder blocks. these blocks are entirely based on the self-attention mechanism @xcite , eliminating the sequential bottleneck of previous recurrent models. bert models are pre-trained on two specifically devised language modeling tasks that allow the networks to learn semantically and contextually meaningful representations of text. these models can then be fine-tuned on specific tasks quite easily (in the case of classification, by adding a simple linear layer as a classifier) to obtain stateof-the-art performances. in our case, we utilize the models available on huggingface @xcite , which are pre-trained on bookcorpus @xcite and english wikipedia, and have a vocabulary of around 30,500 word segments. ## experiments we report in this section the experiments we conducted to select the most suitable summarization strategy and to determine the effectiveness of the multi-level classifiers. while the core of our experiments was performed on bert's base pre-trained model (i.e., "bert-base-uncased"), we also report results using a larger bert lm (i.e., "bert-largeuncased"). ## conclusion in this article, we experimented with contextualized lms for tic, and found that different document embedding summarization strategies are a major factor in classification performance. moreover, we devised two multi-level classification approaches based on lms, and found further improvement by injecting information from the label hierarchy within the architecture. we hope our work can provide useful insights into the usage of bert models for classification in a previously understudied domain.