Evaluating Inter-Bilingual Semantic Parsing for I ndian Languages (2023)

Abstract: Despite significant progress in Natural Language Generation for Indian languages (IndicNLP), there is a lack of datasets around complex structured tasks such as semantic parsing. One reason for this imminent gap is the complexity of the logical form, which makes English to multilingual translation difficult. The process involves alignment of logical forms, intents and slots with translated unstructured utterance. To address this, we propose an Inter-bilingual Seq2seq Semantic parsing dataset IE-SemParse Suite for 11 distinct Indian languages. We highlight the proposed task’s practicality, and evaluate existing multilingual seq2seq models across several train-test strategies. Our experiment reveals a high correlation across performance of original multilingual semantic parsing datasets (such as mTOP, multilingual TOP and multiATIS++) and our proposed IE-SemParse suite.

## introduction task-oriented parsing (top) is a sequence to sequence (seq2seq) natural language understanding (nlu) task in which the input utterance is parsed into its logical sequential form. refer to figure 1 where logical form can be represented in form of a tree with intent and slots as the leaf nodes @xcite . with the development of seq2seq models with self-attention @xcite , there has been an upsurge in research towards developing generation models for complex top tasks. such models explore numerous training and testing strategies to further enhance performance @xcite . most of the prior work focus on the english top settings. however, the world is largely multilingual, hence new conversational ai systems are also expected to cater to the non-english speakers. in that regard works such as mtop @xcite ., * equal contribution inter-bilingual top english monolingual top [in:create_timer [sl:method_timer timer] [sl:date_time for 20 minutes]] set a timer for twenty minutes seq2seq parsing seq2seq parsing [in:create_timer [sl:method_timer timer] [sl:date_time for 20 minutes]] बीस िमनट क े िलए टाइमर से ट कर seq2seq parsing [in:create_timer [sl:method_timer टाइमर] [sl:date_time बीस िमनट क े िलए]] ## why inter bilingual parsing? in this section, we delve deeper into the advantages of our inter-bilingual parsing approach and how it affects the dialogue management and response generation. we will address the question: "why preserve english slot values in the logical form?". limited decoder vocabulary: using only english logical forms simplifies the seq2seq model decoder by reducing its vocabulary to a smaller set. this will make the training process more stable and reduce the chances of hallucination which often occurs in decoders while decoding long sequences with larger vocabulary size @xcite . ## ie-semparse creation and validation in this section, we describe the ie-semparse creation and validation process in details. ## experimental evaluation for our experiments, we investigated into the following five train-test strategies: 1. indic train: models are both finetuned and evaluated on indic language. 2. english+indic train: models are finetuned on english language and then indic language and evaluated on indic language data. 3. translate test: models are finetuned on english data and evaluated on back-translated english data. 4. train all: models are finetuned on the compound dataset of english + all other 11 indic languages and evaluated on indic test dataset. 5. unified finetuning: indicbart-m2o and mbartlarge-50-m2o models are finetuned on all three datasets for all eleven languages creating unified multi-genre (multi-domain) semantic parsing models for all 3 datasets for all languages. this can be considered as data-unified extension of 4th setting. models: the models utilized can be categorized into four categories as follows: (a.) mul-tilingual such as mbart-large-50, mt5base such as (b.) indic specific such as in-dicbart (c.) translation prefinetuned such as indicbart-m2o, mbart-large-50-m2o, which are pre finetuned on xx-en translation task (d.) monolingual (english) such as t5-base, t5-large, bart-large, bart-base used only in translate test setting. the models are specified in the table's §8 "hyper parameter" column, with details in the appendix §c. details of the fine-tuning process with hyperparameters details and the model's vocabulary augmentation are discussed in the appendix §d and §e respectively. evaluation metric: for evaluation, we use tree labelled f1-score for assessing the performance of our models from the original top paper @xcite . this is preferred over an exact match because the latter can penalize the model's performance when the slot positions are out of order. this is a common issue we observe in our outputs, given that the logical form and utterance are not in the same language. however, exact match scores are also discussed in appendix §f.5. ## related work multi-lingual semantic parsing: recently, top has attracted a lot of attention due to the development of state-of-the-art seq2seq models such as bart @xcite and t5 @xcite . moreover, several works have extended top to the multilingual setting, such as mtop, multilingual-top, and multi-atis++. the recent massive dataset @xcite covers six indic languages vs eleven in our work, and only contains a flat hierarchical structure of semantic parse. furthermore, the logical form annotations in massive are not of a similar format to those in the standard top dataset. 10 for 11 x 11 pairs indicnlp: some works have experimented with code-mixed hindi-english utterances for semantic parsing tasks, such as cst5 @xcite . in addition to these advances, there have been significant contributions to the development of indicspecific resources for natural language generation and understanding, such as indicnlg suite @xcite , indicbart @xcite , and indicglue @xcite . also, some studies have investigated the intra-bilingual setting for multilingual nlp tasks, such as indicxnli @xcite and ei-infotabs @xcite . in contrast to prior works, we focus on the complex structured semantic parsing task. llms and zero shot: our work is also related to zero-shot cross-lingual @xcite and cross-domain @xcite semantic parsing, which aims to parse utterances in unseen languages or domains. moreover, recent methods use scalable techniques such as automatic translation and filling @xcite and bootstrapping with llms @xcite @xcite to create semantic parsing datasets without human annotation. unlike previous methods such as translate-align-project (tap) @xcite and translate and fill (taf) @xcite , which generate semantic parses of translated sentences, they propose a novel approach that leverages llms to generate semantic parses of multilingual utterances. ## conclusion and future work we present a unique inter-bilingual semantic parsing task, and publish the ie-semparse suite, which consists of 3 inter-bilingual semantic parsing datasets for 11 indic languages. additionally, we discuss the advantages of our proposed approach to semantic parsing over prior methods. we also analyze the impact of various models and train-test procedures on ie-semparse performance. lastly, we examine the effects of variation in logical forms and languages on model performance and the correlation between languages. for future work, we plan to release a sota model, explore zero-shot parsing @xcite , enhance ie-semparse with human translation @xcite , explore zeroshot dataset generation @xcite , leverage llm for scalable and diverse dataset generation @xcite , and evaluate instruction fine-tuning models.