TEXTOIR : An Integrated and Visualized Platform for Text Open Intent Recognition (2021)

Abstract: TEXTOIR is the first integrated and visualized platform for text open intent recognition. It is composed of two main modules: open intent detection and open intent discovery. Each module integrates most of the state-of-the-art algorithms and benchmark intent datasets. It also contains an overall framework connecting the two modules in a pipeline scheme. In addition, this platform has visualized tools for data and model management, training, evaluation and analysis of the performance from different aspects. TEXTOIR provides useful toolkits and convenient visualized interfaces for each sub-module, and designs a framework to implement a complete process to both identify known intents and discover open intents.

## introduction analyzing user intents plays a critical role in human-machine interaction services (e.g., dialogue systems). however, many current dialogue systems are confined to recognizing user intents in closedworld scenarios, and they are limited to handle the uncertain open intents. as shown in figure 1 , it is easy to identify specific purposes, such as flight booking and restaurant reservation. nevertheless, as the user intents are varied and uncertain, predefined categories may be insufficient to cover all user needs. that is, there may exist some unrelated user utterances with open intents. it is valuable to distinguish these open intents from known intents, which is helpful to improve service qualities, and further discover fine-grained classes for mining potential user needs. we divide open intent recognition (oir) into two modules: open intent detection and open intent discovery. the first module aims to identify n-class known intents and detect one-class open intent @xcite @xcite . it can identify known classes but fail to discover specific open classes. the second module further groups the one-class open intent into multiple fine-grained intent-wise clusters @xcite @xcite . nevertheless, the adopted clustering techniques are not able to identify known categories. the two modules have achieved huge progress with various advanced methods on benchmark datasets. however, there still exist some issues, which bring difficulties for future research. firstly, there are no unified and extensible interfaces to integrate various algorithms for two modules, bringing challenges for further model development. secondly, the current methods of the two modules lack convenient visualized tools for model management, training, evaluation and result analysis. thirdly, the two modules both have some limitations for oir. that is, neither of them can identify known intents and discover open intents simultaneously. therefore, oir remains at the theoretical level, and it needs an overall framework to connect the two modules for finishing the whole process. to address these issues, we propose textoir, the first integrated and visualized text open intent recognition platform. the platform has the following features: (1) it provides toolkits for open intent detection and open intent discovery, respectively. the toolkits contain flexible interfaces for data, configuration, backbone and method integration. specifically, it integrates a series of advanced models for two modules. each module supports a complete workflow, including data and backbone preparation with different assigned parameters, training, and evaluation. it provides standard and convenient modules to add new methods. more detailed information can be found on https://github.com/ thuiar/textoir . ## open intent recognition platform figure 2 shows the architecture of the proposed textoir platform, which contains four main modules. the first module integrates a series of standard benchmark datasets. the second and third modules have toolkits for both open intent detection and open intent discovery. besides, it visualizes the whole process (including model management, training, evaluation and result analysis) of two modules. the last module leverages the two modules in a pipeline framework to finish open intent recognition. ## pipeline framework the two modules of open intent detection and discovery are closely related. however, there lacks an overall framework to successively invoke the two modules for both identifying known intents and discovering open intents. textoir addresses this issue with a proposed pipeline framework, as shown in figure 3 and figure 4 . the pipeline framework first processes the original data for two modules. then, it feeds the labeled known intent data to the open intent detection module and trains the selected model by the users. as there is still a mass of unlabeled data containing both known and open intents, it leverages the welltrained open intent detection model to predict the unlabeled training data. the evaluated results on training data contain identified known intents and the detected open intent. we use the predicted known intent data, detected open intent and original labeled data as the inputs of the open intent discovery module. in this case, the discovery module benefits from the detection module to obtain the augmented inputs for training. next, the preferred clustering method selected by the users is trained to obtain the open intent clusters. after training the two modules, they are used to perform open intent recognition on unlabeled data. specifically, the well-trained open intent detection method is first used to predict the identified known intents and detected open intent. then, the open intent discovery method is utilized to predict the detected open intent data to obtain the finegrained open intent clusters. finally, the keybert toolkit (mentioned in section 4.2.2) is leveraged to extract keywords for each open intent cluster with similar-intent sentences. therefore, our framework identifies known intents and discovers open intent samples in group with keywords as recommended labels. ## experiments we use four intent benchmark datasets mentioned in section 2.1 to verify the performance of our tex-toir platform. the known intent ratios are varied between 25%, 50% and 75%. the labeled proportions are varied between 50% and 100%. to evalu1 . the pipeline framework successfully connects two modules, and achieves competitive and robust results in different settings. it essentially overcomes the shortcoming of two modules, and uses the first module to identify known intents, the second module to discover open intents. ## conclusion we propose the first open intent recognition platform textoir, which integrates two complete modules: open intent detection and open intent discovery. it provides toolkits for each module with common interfaces and integrates multiple advanced models and benchmark datasets for the convenience of further research. additionally, it realizes a pipeline framework to combine the advantages of two modules. the overall framework achieves both identifying known intents and discovering open intents. a series of visualized surfaces help users to manage, train, evaluate, and analyze the performance of different methods.