Intent Detection and Slot Filling for Home Assistants: Dataset and Analysis for B angla and S ylheti (2023)

Abstract: As voice assistants cement their place in our technologically advanced society, there remains a need to cater to the diverse linguistic landscape, including colloquial forms of low-resource languages. Our study introduces the first-ever comprehensive dataset for intent detection and slot filling in formal Bangla, colloquial Bangla, and Sylheti languages, totaling 984 samples across 10 unique intents. Our analysis reveals the robustness of large language models for tackling downstream tasks with inadequate data. The GPT-3.5 model achieves an impressive F1 score of 0.94 in intent detection and 0.51 in slot filling for colloquial Bangla.

## introduction smart devices have become commonplace, establishing home assistants as indispensable fixtures in contemporary households. these voice-activated virtual companions adeptly manage an array of tasks, ranging from setting reminders to controlling room temperatures. the efficacy of home assistants in performing these tasks is closely intertwined with their underlying natural language understanding (nlu) models, which enable seamless interactions in high-resource languages @xcite @xcite . however, this advantage in nlu capabilities is not extended to low-resource languages @xcite , presenting a notable discrepancy. this discrepancy holds considerable significance, especially considering the global demand for home assistants and the extensive usage of low-resource languages, which have a substantial speaker base. bangla and sylheti @xcite , with 285 million native speakers combined, have rich cultural and colloquial nuances. specialized datasets are needed to capture these intricacies as users prefer to interact with home assistants in their native languages, highlighting the research need @xcite . the language understanding of home assistants is dependent on two key nlu tasks: intent detection and slot filling @xcite . intent detection determines user actions, like playing music or checking the weather, while slot filling extracts specific details, such as song titles or locations. these tasks enable seamless human-device interactions, especially for home assistants. research on intent detection and slot filling primarily focuses on high-resource languages @xcite @xcite . while there have been limited studies dedicated to the bangla language @xcite @xcite , none of them have addressed the tasks of intent detection and slot filling in bangla. furthermore, these studies have not taken into account colloquial variants or closely related languages like sylheti. this gap in research leaves a significant portion of the speaker base underserved. this paper bridges this research gap with several notable contributions. firstly, we introduce a comprehensive dataset encompassing 328 entries for intent detection and slot filling for each of the three languages -totaling 984 samples. these languages include formal bangla, colloquial bangla, and colloquial sylheti. we further show a comparative study between generative llms and state-of-theart language models for intent detection and slot filling. ## dataset at the core of our exploration stands a meticulously curated dataset that is inspired by the snips dataset @xcite , which caters to the broad audience. ## methodology and experimental setup our experiments were divided into four phases. in our initial experiment, we employed jointbert @xcite , the state-of-the-art model in this domain, for both intent detection and slot-filling tasks. in our next experiment, jointbert was retained for intent detection, while we explored the capabilities of gpt-3.5 (generative pre-trained transformer) (brown et al., 2020) model for slot filling. the third experiment fully utilized gpt-3.5 for both tasks. for our concluding experiment, we provided gpt-3.5 with the original intents and then analyzed its performance on the slot-filling task. the final experiment gives the raw result of slot-filling for the gpt model. jointbert leverages the bert @xcite model to provide a unified approach encompassing both intent classification and slot filling by utilizing the representations from the pre-trained bert model. we employed the default bert tokenizer and maintained consistent parameters for all three languages. the utilization of these default settings and tokenization methods ensures an equitable and consistent evaluation across the languages. gpt-3.5 (generative pre-trained transformer) (brown et al., 2020) model operates on the transformer architecture and is adept at generating text resembling human language by predicting subsequent words or tokens in a sequence. gpt-3.5's deep contextual understanding is a result of extensive pre-training on a diverse corpus of textual data, encompassing various languages and linguistic intricacies enabling it to excel across a spectrum of nlp tasks @xcite @xcite . we used gpt in a few-shot setting, passing 5 training samples along with the prompt. rigorous prompt engineering was performed before settling on the two prompts for the two tasks. figure 3 and figure 4 show the final versions of the prompts used in the experimentations. ## results tables 2 and 3 present the performance of the models we evaluated on our intent detection and slotfilling tasks. a clear pattern emerges: gpt-3.5 consistently outperforms jointbert in both tasks. in handling the complexities of this task. a significant reason behind gpt-3.5's superior performance is its broader exposure to diverse languages during training, including bangla. joint-bert, conversely, hasn't been specifically trained on any bangla dataset. this linguistic familiarity gives gpt-3.5 a clear advantage, enabling it to process and interpret bangla's nuances far more effectively than jointbert. the results underline the significance of using llms for low-resource languages, especially in scenarios where obtaining high volumes of training data for a particular downstream task is challenging. ## conclusion in the era of smart devices, a home assistant's voice interfaces must resonate with the authentic linguistic intricacies of its users. our research presents the first-ever dataset for intent detection and slot filling in bangla and sylheti, emphasizing their colloquial forms. this focus on colloquial forms bridges the often-overlooked gap between formal language models and the nuances of everyday speech. by championing colloquial forms, we ensure a voice interface that's more natural and attuned to genuine communication habits. through rigorous data collection and validation, we have produced a highquality benchmark dataset, providing a solid foundation for subsequent analyses and model evaluations. the comparative study between large lan-guage models (llm) like gpt-3.5 and non-llms underscores the remarkable capability of llms to excel even with minimal datasets, marking a considerable stride for underrepresented languages.