Benchmarking Language-agnostic Intent Classification for Virtual Assistant Platforms (2022)

Abstract: Current virtual assistant (VA) platforms are beholden to the limited number of languages they support. Every component, such as the tokenizer and intent classifier, is engineered for specific languages in these intricate platforms. Thus, supporting a new language in such platforms is a resource-intensive operation requiring expensive re-training and re-designing. In this paper, we propose a benchmark for evaluating language-agnostic intent classification, the most critical component of VA platforms. To ensure the benchmarking is challenging and comprehensive, we include 29 public and internal datasets across 10 low-resource languages and evaluate various training and testing settings with consideration of both accuracy and training time. The benchmarking result shows that Watson Assistant, among 7 commercial VA platforms and pre-trained multilingual language models (LMs), demonstrates close-to-best accuracy with the best accuracy-training time trade-off.

## introduction virtual assistant (va) platforms that enable customers to train and deploy their chatbots have seen growing demand in recent years. this has attracted significant interest from both industry and academia to develop new machine learning (ml) models and datasets for these task-oriented dialog systems. in a dialog system, intent classification as the core component identifies user intent of a user's utterance so that the system can respond appropriately by triggering dialog nodes in predefined dialog trees. although there has been a lot of exploration around implementing intent classification models for english, not much work has been extended to low-resource languages. due to the vast number of world languages, it is not trivial for an enterprise va platform to support its global customers. ## benchmarking in this section, we firstly introduce the three benchmarking settings in our experiments, and then describe the va platforms and models we evaluate. lastly, we present and analyze the results. ## results and analysis standard training setting ## conclusion in this paper, we propose a robust evaluation framework to benchmark 7 intent classification models in multiple languages. on 9 public datasets and 20 internal datasets covering 10 languages. the benchmark results show that while labse produces the highest accuracy in almost all evaluation settings, watson assistant achieves competitive performance with much less cost of training time and resource. the large lms does not always outperform the models that only need cpus. through our work, we hope to encourage more research and development on language-agnostic chatbot solutions.