Cross-lingual Text-to- SQL Semantic Parsing with Representation Mixup (2022)

Abstract: We focus on the cross-lingual Text-to-SQL semantic parsing task,where the parsers are expected to generate SQL for non-English utterances based on English database schemas.Intuitively, English translation as side information is an effective way to bridge the language gap,but noise introduced by the translation system may affect parser effectiveness.In this work, we propose a Representation Mixup Framework (Rex) for effectively exploiting translations in the cross-lingual Text-to-SQL task.Particularly, it uses a general encoding layer, a transition layer, and a target-centric layer to properly guide the information flow of the English translation.Experimental results on CSpider and VSpider show that our framework can benefit from cross-lingual training and improve the effectiveness of semantic parsers, achieving state-of-the-art performance.

## introduction the task of semantic parsing is to translate natural language utterances into meaning representations, such as lambda calculus @xcite or a programming language @xcite @xcite . more recently, textto-sql semantic parsing, using sql queries as the meaning representation, has attracted increasing attention from both academia and industry researchers @xcite @xcite @xcite . benefiting from recently annotated large-scale datasets @xcite , research in text-to-sql has been greatly expedited. moreover, due to the development of encoderdecoder pre-trained models @xcite , semantic parsers have been improved significantly, benefiting from contextualized representations @xcite . however, these advances have been figure 1 : an illustration of the cross-lingual text-to-sql task. the utterance in english: "find the government form name and total population for each government form whose average life expectancy is longer than 72.". the automatic translation fails to translate "trung bình" in vietnamese into "average" in english. another direction for solving the cross-lingual text-to-sql problem is to build target language annotated datasets and train a target language parser directly @xcite . however, these methods fail to leverage knowledge from english parsers (learned from annotated english data), which has the potential to benefit non-english text-to-sql parsing. in this work, we propose rex, a representation mixup framework for cross-lingual modeling that utilizes both the english data and the annotated data in the target language. first, rex adopts a twostage training strategy where the target language models are first initialized with the pre-trained english parser and then trained with the target language data. using this method, basic schema encoding ability and sql decoding ability of the english parsers can be reused during target language training. second, to further make use of english parsers' utterance encoding ability, we use english translations as context augmentation for bridging the cross-lingual gap and facilitating non-english model training. instead of simply concatenating the english translation and the target utterance, rex takes a general encoder, a transition layer, and a targetcentric encoder to properly guide information flow of english translations, to mitigate the aforementioned issue around noisy translations. in detail, the general encoder generates contextual representations for bilingual utterances and database schemas. the transition layer is leveraged to obtain a crosslingual mixup representation of the target language utterance, aiming to make the best use of english translations while minimizing the noise introduced. lastly, the target-centric encoder focuses on the interaction between the target language utterance and the source language schema, by ignoring the side effects caused by translations. we test our rex framework on two non-english text-to-sql semantic parsing datasets, cspider and vspider. experimental results on the benchmarks show that our framework can further improve upon simple yet effective baselines. at the time of publication, rex achieves state-of-the-art performance on the cspider leaderboard based on the results on the hidden test set. our contributions are summarized as follows: ## problem definition and notation given an utterance @xmath0, x 2 , ..., x n ) and a database schema s, a text-to-sql model is expected to translate the utterance into a valid sql query. our framework is based on a standard encoder-decoder architecture. for the task, we assume the existence of an internationalized database where the database schema s is in english. the natural language queries from users are not in english; we denote these non-english languages as target languages. formally, we denote x @xmath1, x 2 , ..., x nt } as an utterance in the target language with n t tokens. similarly, the utterance in the source language foot_1 with n s tokens is denoted x @xmath2, x 2 , ..., x ns }. we assume the database schema s contains several tables @xmath3, c 2 , ..., c |t | }, where |t | denotes the number of columns in table t . ## baseline: single-source input here we introduce a sequence-to-sequence based semantic parsing model and a baseline that leverages english translation for non-english text-to-sql semantic parsing. the model for text-to-sql semantic parsing has been continuously improved in recent years @xcite @xcite @xcite @xcite @xcite . specifically, @xcite utilize the pre-trained sequence-to-sequence model transition layer sql decoder general encoder target-centric encoder en utterance target utterance schema select sum(population).. multi-head attention multi-head attention multi-head attention ln ln add add ln + ln & ff & add en utterance target utterance t5 as the parser to directly generate sql, simplifying the intermediate representation design for the grammar-based decoder. in detail, the model input is the concatenation of utterance x and linearized database schema s. with a pre-trained encoder, the contextualized utterance representation h x and the database schema representation h d can be obtained. the pre-trained decoder leverages these contextualized hidden states for generating sql in an autoregressive fashion with constrained decoding. this architecture obtains state-of-the-art performance on english benchmarks without complex modeling. by replacing the t5 model with its multilingual version, mt5 @xcite , the model can be applied to non-english training data, to obtain parsers that have the ability to handle non-english utterances. we denoted this as single-source target-language training. ## representation mixup framework here, we propose a representation mixup framework for cross-lingual text-to-sql semantic parsing. as shown in figure 2 , the encoder stacks several general encoding layers with a transition layer, and some target-centric encoding layers. first, utterances from the source and target languages can be encoded separately or jointly in the general encoder layers. then, a transition layer implements the representation mixup between the input sequences, such as source language utterance, target language utterance, and database schema, in a specific layer. then, a target-centric encoding layer will try to ignore the noise produced by machine translation systems by only focusing on the target language utterance. finally, a sql decoder leverages the information from the encoder component to generate a full sql query. ## framework configurations by configuring parameters such as m and k, some basic model architectures can be obtained. multi-source input with concatenation: concatenation is a simple yet effective baseline for leveraging both the source language utterance and the target language utterance at the same time. by setting @xmath4, our framework is configured as a simple concatenation model. in this case, the decoder can leverage bilingual information to generate the sql query. focused simple concatenation: by setting @xmath5, our rex framework can obtain a new architecture, denoted as focused simple concatenation. different from simple concatenation, the sql decoder only focuses on the target utterance component and the database schema, ignoring the source language utterance. this can reduce the negative effects caused by noisy machine translations. ## experimental setup datasets: in this work, we evaluate our framework on two text-to-sql semantic parsing datasets, cspider @xcite and vspi-der @xcite , which are chinese and vietnamese cross-domain text-to-sql datasets adapted from the spider benchmark @xcite . for cspider, we use exact set match (em) accuracy as the evaluation metrics. for vspider, we use both em accuracy and test-suite (ts) @xcite ) accuracy for evaluation. for exact set match accuracy, the prediction is classified as correct only if all of the components (select clause, where clause, having clause, etc.) are correct. the test-suite accuracy, which is an improved version of execution accuracy (if the execution results of a predicted sql query are the same as those of the ground truth sql query, then it is classified as correct), serves as a tight upper bound for semantic accuracy @xcite . model training: our rex framework is an adapted sequence-to-sequence transformer. benefiting from pre-trained sequence-to-sequence language models such as bart @xcite or t5 @xcite , performance is significantly improved by finetuning pre-trained models. furthermore, because our rex framework is expected to process utterances in multiple languages, our experiments are based on mt5-large, which has 24 layers. to leverage english annotated data, we conduct two-stage training: we first train mt5-large on the english dataset to obtain a trained parser checkpoint. note that this trained english parser is based on a standard sequence-to-sequence architecture instead of the rex framework. the input of the parser is the concatenation of the english utterance comparing the multi-source models to singlesource ones, we observe that the extra information can improve parser effectiveness. with multisource concatenation, the parser is already competitive with the state-of-the-art parser. our rex framework further improves the model by 0.6% over the multi-source input with concatenation. vspider: because our data split on vspider is different from tuan nguyen et al. (2020), our results are not directly comparable. foot_2 the main results for vspider are shown in ## discussion to investigate issues in cross-lingual semantic parsing and to better understand our rex framework, we performed several ablation experiments. ## conclusions we propose the rex framework that effectively integrates information from english translations into the modeling of target language utterances. more specifically, we propose three different transition layer implementations that enhance the interactions among different components. we further compare their effectiveness with detailed ablation studies. experiments show that our framework is robust to translation noise by controlling the information flow properly, outperforming existing baselines on the vspider and cspider benchmarks.