A Pilot Study of Text-to- SQL Semantic Parsing for V ietnamese (2020)

Abstract: Semantic parsing is an important NLP task. However, Vietnamese is a low-resource language in this research area. In this paper, we present the first public large-scale Text-to-SQL semantic parsing dataset for Vietnamese. We extend and evaluate two strong semantic parsing baselines EditSQL (Zhang et al., 2019) and IRNet (Guo et al., 2019) on our dataset. We compare the two baselines with key configurations and find that: automatic Vietnamese word segmentation improves the parsing results of both baselines; the normalized pointwise mutual information (NPMI) score (Bouma, 2009) is useful for schema linking; latent syntactic features extracted from a neural dependency parser for Vietnamese also improve the results; and the monolingual language model PhoBERT for Vietnamese (Nguyen and Nguyen, 2020) helps produce higher performances than the recent best multilingual language model XLM-R (Conneau et al., 2020).

## introduction semantic parsing is the task of converting natural language sentences into meaning representations such as logical forms or standard sql database queries @xcite , which serves as an important component in many nlp systems such as question answering and task-oriented dialogue @xcite @xcite . the significant availability of the world's knowledge stored in relational databases leads to the creation of large-scale text-to-sql datasets, such as wikisql @xcite and spider @xcite , which help boost the development of various state-ofthe-art sequence-to-sequence (seq2seq) semantic parsers @xcite @xcite . compared to wikisql, the spider dataset presents challenges not only in handling complex questions but also in generalizing to unseen databases during evaluation. most sql semantic parsing benchmarks, such as wikisql and spider, are exclusively for english. thus the development of semantic parsers has largely been limited to the english language. as sql is a database interface and universal semantic representation, it is worth investigating the text-to-sql semantic parsing task for languages other than english. especially, the difference in linguistic characteristics could add difficulties in applying seq2seq semantic parsing models to the non-english languages @xcite . for example, about 85% of word types in vietnamese are composed of at least two syllables @xcite . unlike english, in addition to marking word boundaries, white space is also used to separate syllables that constitute words in vietnamese written texts. for example, an 8-syllable written text "có bao nhiêu quốc gia ở châu âu" (how many countries in europe) forms 5 words "có bao_nhiêu how many quốc_gia country ở in châu_âu europe ". thus it is interesting to study the influence of word segmentation in vietnamese on its sql parsing, i.e. syllable level vs. word level. in terms of vietnamese semantic parsing, previous approaches construct rule templates to convert single database-driven questions into meaning representations (nguyen and @xcite @xcite @xcite . recently, @xcite formulate the textto-sql semantic parsing task for vietnamese as a sequence labeling-based slot filling problem, and then solve it by using a conventional crf model with handcrafted features, due to the simple structure of the input questions they deal with. note that seq2seq-based semantic parsers have not yet been explored in any previous work w.r.t. vietnamese. semantic parsing datasets for vietnamese include a corpus of 5460 sentences for assigning semantic roles @xcite and a small textto-sql dataset of 1258 simple structured questions over 3 databases @xcite . however, these two datasets are not publicly available for research community. in this paper, we introduce the first public largescale text-to-sql dataset for the vietnamese semantic parsing task. in particular, we create this dataset by manually translating the spider dataset into vietnamese. we empirically evaluate strong seq2seq baseline parsers editsql @xcite and irnet @xcite on our dataset. extending the baselines, we extensively investigate key configurations and find that: (1) our human-translated dataset is far more reliable than a dataset consisting of machine-translated questions, and the overall result obtained for vietnamese is comparable to that for english. (2) automatic vietnamese word segmentation improves the performances of the baselines. (3) the npmi score @xcite is useful for linking a cell value mentioned in a question to a column in the database schema. (4) latent syntactic features, which are dumped from a neural dependency parser pre-trained for vietnamese (nguyen and verspoor, 2018), also help improve the performances. (5) highest improvements are accounted for the use of pre-trained language models, where phobert (nguyen and nguyen, 2020) helps produce higher results than xlm-r @xcite . we hope that our dataset can serve as a starting point for future vietnamese semantic parsing research and applications. we publicly release our dataset at: https://github.com/ vinairesearch/vitext2sql . ## our dataset we manually translate all english questions and the database schema (i.e. table and column names as well as values in sql queries) in spider into vietnamese. note that the original spider dataset consists of 10181 questions with their corresponding 5693 sql queries over 200 databases. however, only 9691 questions and their corresponding 5263 sql queries over 166 databases, which are used for training and development, are publicly available. thus we could only translate those available ones. the translation work is performed by 1 nlp researcher and 2 computer science students (ielts otherwise, for complex questions, we will rephrase them based on the semantic meaning of the corresponding sql queries to obtain the most natural language questions in vietnamese. ## conclusion in this paper, we have presented the first public large-scale dataset for vietnamese text-to-sql semantic parsing. we also extensively experiment with key research configurations using two strong baseline models on our dataset and find that: the input representations, the npmi-based approach for schema linking, the latent syntactic features and the pre-trained language models all have the influence on this vietnamese-specific task. we hope that our dataset can serve as the starting point for further research and applications in vietnamese question answering and dialogue systems.