Effect of Pivot Language and Segment-Based Few-Shot Prompting for Cross-Domain Multi-Intent Identification in Low Resource Languages (2023)

Abstract: NLU (Natural Language Understanding) has considerable difficulties in identifying multiple intentions across different domains in languages with limited resources. Our contributions involve utilizing pivot languages with similar semantics for NLU tasks, creating a vector database for efficient retrieval and indexing of language embeddings in high-resource languages for Retrieval Augmented Generation (RAG) in low-resource languages, and thoroughly investigating the effect of segmentbased strategies on complex user utterances across multiple domains and intents in the development of a Chain of Thought Prompting (COT) combined with Retrieval Augmented Generation. The study investigated recursive approaches to identify the most effective zeroshot instances for segment-based prompting. A comparison analysis was conducted to compare the effectiveness of sentence-based prompting vs segment-based prompting across different domains and multiple intents. This research offers a promising avenue to address the formidable challenges of NLU in low-resource languages, with potential applications in conversational agents and dialogue systems and a broader impact on linguistic understanding and inclusivity.

## introduction cross-domain multi-intent classification is a crucial task in natural language understanding with extensive applications in conversational agents and dialogue systems.for low resource language, this presents an intricate and pressing challenge.we conducted our research specifically on indian languages, with bengali being identified as a language with limited resources.this paper unveils a comprehensive methodology that leverages high-resource languages as a pivot language.the foundation of our approach relies on discovering high-resource languages that closely resemble the semantics of their low-resource equivalents.this study utilizes the notion of retrieval-augmented generation on language models (llms) to examine the impact of using a pivot language that is semantically comparable and selecting a small number of instances from high-resource languages to build context.the significance of storing the embeddings in the vector database has also been investigated.further experiments in the research showed how the effect of chunking of cross-domain, multi-intent userutterances affects the results from llms.various experiments conducted as a part of the research offer a key to unlocking the complex semantics of low-resource languages. in essence, our research journey optimizes intent classification and domain categorization in low-resource languages, rendering it more inclusive, efficient, and adaptable to linguistic nuances and unexplored endangered lowresource languages. overall , we make the following contributions: ## related work in the low-resource language context, cross-lingual multi-intent and multi-domain extraction remains underexplored, notably for indian languages like bengali. our research aims to bridge these gaps by employing a chain of thought prompting and retrieval-augmented generation utilizing a vector database. we use a few shot examples from a corresponding hrl to generate a context and use it as a prompt. we have also explored the effect of the choice of few shot examples for each chunk from a user utterance. for low-resource languages, prompting has been proven to be a promising technique and has been shown to give better results. in @xcite , the authors introduced relationprompt, a zero-shot relation extraction approach which utilizes prompts to generate synthetic relation examples, which are then used to train a relation extractor for low-resource languages. @xcite further enhances multilingual capabilities by appointing cross-lingualthought prompting (xlt). however, the substantial requirement of computational resources led the way to further research. for intent classification in data-scarce environments, @xcite explores zero-shot and few-shot techniques which utilize external knowledge, such as word embeddings or taxonomies, to transfer knowledge between similar tasks. @xcite further investigates the impact of prompt order on the performance of few-shot learning in large language models. however, their focus on english does not cater to the complexity of languages like bengali. these lead the way to retrieval-augmented generation and vector databases to address these challenges. our approach combines insights from @xcite and @xcite to predict multiintent and multi-domain with high-resource references and enhance performance in low-resource languages through data augmentation. @xcite outlines an approach to augment the context of prompts with semantically similar sentences retrieved from a high-resource language that enhances nlu models for low-resource languages, offering insights into effective retrieval-augmented prompts. @xcite investigates the gender bias of chatgpt across six low-resource languages, including bengali. this underlines the importance of mitigating biases in ai systems. in @xcite , the authors concluded that hallucinations are a significant challenge for large multilingual translation models and recommended that developers of multilingual translation systems should be aware of the issue of hallucinations and take steps to mitigate it. @xcite utilizes pretrained language models (plms) to learn text representation and generation capabilities on a largescale unsupervised tibetan corpus. @xcite emphasizes parameter-efficient prompt learning, aligning with our intent and domain prediction goals. however, they may not be tailored to the cross-lingual and multi-intent prediction that we aim to incorporate. @xcite further introduces domain-aligned data augmentation, crucial for accurate cross-lingual intent and domain prediction in low-resource languages. it may not focus on the chain of thought prompting and vector databases, which we consider integral to our approach. ## dataset for retrieving semantically similar documents, wikipedia articles pertaining to the freedom movement of indian have been extracted for indian genre languages -hindi, bengali, marathi, telugu, and punjabi. for retrieval augmented generation, a massive dataset is used. massive is a parallel dataset @xmath0. utterances span 60 intents and include 55 slot types. massive was created by localizing the slurp dataset, composed of general intelligent voice assistant single-shot interactions. for testing the proposed methodology, a test dataset comprising cross-domain and multiple intent user-utterance was used. ## methodology this section focuses on the approach for crossdomain multi-intent identification for low-resource languages using high-resource languages as a pivot language.the approach is essentially divided into four primary components : (1) identifying a high-resource language (hrl) that is semantically similar to a low-resource language (lrl) within a language family. (2) implementation of a vector database to store embeddings for the pivot language. (3) developing a systematic approach to determine the appropriate number of documents per chunk for retrieval augmented generation in the context of intricate user-utterances that span across many domains and intentions, requiring segmentation into smaller chunks. ( 4 ) conducting a comparison analysis to determine the accuracy and efficiency of segment-based prompting against sentence-based prompting (where a complete sentence is provided as a prompt without breaking it into smaller parts). for carrying out the experiments, text-embedding-ada-002 is used for extracting embeddings, and gpt 3.5 turbo/gpt 4 is used as the large language model. cosine similarity is used as a similarity measure for computing similarity between documents. langchain is used as an orchestrator to manage llms. open source in-memory chromadb is used as a vector database for storing embeddings. the architecture diagram of the proposed methodology is given in figure 1 . ## results and discussion the implementation of the model utilized langchain, with chromadb serving as the vector database. the libraries developed by openai were heavily utilized, and the specific model selected for this experiment was openai's gpt-3.5-turbo. ## conclusion this study presents the concept of cross-domain multi-intent identification for low-resource languages by utilizing a pivot high-resource language. we have conducted our experiments on indian regional languages. we utilized segment-based retrieval augmented generation by employing a vector database in conjunction with chain of thought prompting, resulting in enhanced accuracy and efficiency. the embeddings for languages with abundant resources were put in the vector database to facilitate efficient retrieval and indexing. we evaluated the necessity of splitting a complex sentence that spans many domains into segments and performing rag (retrieval-augmented generation) with various k values. here, k refers to a few-shot examples in a high-resource language which are semantically correlated, and these examples are used to build contexts for prompts used in chain of thought prompting. future research could primarily concentrate on extending this study to various nlp tasks and multiple low-resource endangered unknown languages. additionally, it should prioritize the exploration of various ways to identify the k most ideal values for few-shot prompting in low-resource languages.