Intent Segmentation of User Queries Via Discourse Parsing (2020)

Abstract: In this paper, we explore a new approach based on discourse analysis for the task of intent segmentation. Our target texts are user queries from a real-world chatbot. Our results show the feasibility of our approach with an F1-score of 82.97 points, and some advantages and disadvantages compared to two machine learning baselines: BERT and LSTM+CRF.

## introduction chatbots, a type of dialogue system, have been widely deployed for commercial applications, such as flight booking and troubleshooting support. one key step of frame-based chatbots is to extract the intent of the user, i.e. to recognize the goal of the user's query @xcite . while the task of single-intent prediction, when a query is assumed to encode only one intent, can be tackled as a multi-class classification problem @xcite , a more complex scenario arises when a query encodes more than two intents-a multi-intent query. in this case, a multi-class classification strategy is applicable if single-intent labels are combined together to form new intent labels; however, this strategy may suffer from data sparsity problems @xcite . another option is to treat multi-intent prediction as a multi-label task where binary classifiers are trained for each singleintent label, and so an arbitrary number of singleintent labels can be predicted for any given query; however, this strategy may not yield the best accuracy results @xcite . a third strategy is to segment a user query into intent segments where each segment encodes a single-intent label; these segments then are passed to a pre-trained intent classifier. in this paper we explore this strategy by focusing on the intent segmentation problem. to tackle this problem, we use real-world user queries from our online chatbot moli as our data. moli receives queries from motorola cellphone users related to troubleshooting, warranty and sales issues. we use multi-intent queries which encode from two up to six different intents. each of these intents may have the same importance as the rest, so predicting a single-intent label for a multi-intent query may lead to a poor user experience. this situation motivates us to propose a segmentation method that is able to find the boundaries among intents in a given query; in this way, single-intent classifiers from previous works can be used to predict the intent label for each segment of a query. a possible option to segment a query into intent segments are statistical models previously used for sentence segmentation @xcite . these models have been widely used in dialogue systems to segment a user's utterance into dialogue acts @xcite @xcite , and also for intent segmentation @xcite . even though these models have achieved good results in the literature, in this paper we aim to explore a symbolic approach. to the best of our knowledge, this is the first work where discourse analysis is used for intent-segmentation of user queries for chatbots. previous works in discourse analysis have proposed different ways of structuring texts into symbolic representations, being rst trees the most popular one. an rst parser is a system that seeks to find hierarchical relationships between adjacent discourse units within a text @xcite . to do so, a text is segmented into discourse units and then a tree structure is built by finding the discourse relationships between adjacent discourse units or sub-tree structures; the final tree structure is a representation of the discourse information contained in the text. rst has been widely used in nlp applications such as topic segmentation @xcite , question answering @xcite , sentence compression @xcite and sentiment analysis @xcite . we propose to use an rst parser to segment user queries into intent segments. we aim to answer the research questions: how can we use an rst parser for the task of intent segmentation of user queries? to what extent discourse information is useful for intent segmentation? what are advantages and disadvantages of segmenting queries via an rst parser? we hypothesize that discourse information is helpful to find tokens within a query that are boundaries of intent segments. in other words, we hypothesize that discourse information is helpful to segment a user query into meaningful pieces of text such that each of these pieces encodes a single intent. our results seem to support our hypothesis. also, we compare our approach with two machine learning models, namely bert and lstm+crf. furthermore, we care on the accuracy-speed trade-off of the rst parser for our intent segmenter to be used online; so, we propose a new rst parser that elaborates on top of a well-known and fast parser in the literature, namely hilda @xcite . overall, our results on the task of intent segmentation-above 80% f1-score-show the feasibility of our approach. ## a working-example in this section we provide an example of the problem of segmenting a user query into intent segments. consider the user query in example 1 which describes problems a user has with a cellphone. the gold intent-segmentation of this query is shown in ## results in this section, we report the results of our experiments. the accuracy score of our rst-relation classifier is @xmath0.22% across the 5 rst-relation types on test data. in from the first and third rows of ## discussion and conclusions we worked on the task of intent segmentation of user queries posed to a real-world chatbot. we explored an approach based on discourse analysis 16 we setup the policy that if the number of edus in a query is |eq i | ≤ 7 then we resort to dijkstra search, else we resort to greedy search. we optimized this policy on the development set. 17 an illegal parse is when the rst tree contains at least one illegal sub-tree; for example, the sub-tree (same unit n, (parallel, a, b), c) is illegal because the most internal subtree indicates that edus a and b are parallel, therefore the attempt to concatenate them with edu c results in an illegal action because this implies that a and b must be concatenated beforehand.