A Pilot Study of Text-to-SQL Semantic Parsing for Vietnamese
Abstract
Semantic parsing is an important NLP task.
However, Vietnamese is a low-resource language in this research area.
In this paper, we present the first public large-scale Text-toSQL semantic parsing dataset for Vietnamese.
We extend and evaluate two strong semantic parsing baselines EditSQL (Zhang et al., 2019) and IRNet (Guo et al., 2019) on our dataset.
We compare the two baselines with key configurations and find that: automatic Vietnamese word segmentation improves the parsing results of both baselines; the normalized pointwise mutual information (NPMI) score (Bouma, 2009) is useful for schema linking; latent syntactic features extracted from a neural dependency parser for Vietnamese also improve the results; and the monolingual language model PhoBERT for Vietnamese (Nguyen and Nguyen, 2020) helps produce higher performances than the recent best multilingual language model
XLM-R (Conneau et al., 2020).
Introduction
Semantic parsing is the task of converting natural language sentences into meaning representations such as logical forms or standard SQL database queries (Mooney, 2007), which serves as an important component in many NLP systems such as
Question answering and Task-oriented dialogue (Androutsopoulos et al., 1995; Moldovan et al., 2003; Guo et al., 2018).
The significant availability of the world’s knowledge stored in relational databases leads to the creation of large-scale
Text-to-SQL datasets, such as WikiSQL (Zhong et al., 2017) and Spider (Yu et al., 2018), which help boost the development of various state-ofthe-art sequence-to-sequence (seq2seq) semantic parsers (Bogin et al., 2019; Zhang et al., 2019; Guo et al., 2019).
Compared to WikiSQL, the Spider dataset presents challenges not only in handling complex questions but also in generalizing to unseen databases during evaluation.
Most SQL semantic parsing benchmarks, such as WikiSQL and Spider, are exclusively for English.
Thus the development of semantic parsers has largely been limited to the English language.
As SQL is a database interface and universal semantic representation, it is worth investigating the Text-to-SQL semantic parsing task for languages other than English.
Especially, the difference in linguistic characteristics could add difficulties in applying seq2seq semantic parsing models to the nonEnglish languages (Min et al., 2019).
For example, about 85% of word types in Vietnamese are composed of at least two syllables (Thang et al., 2008).
Unlike English, in addition to marking word boundaries, white space is also used to separate syllables that constitute words in Vietnamese written texts. For example, an 8-syllable written text “Có bao nhiêu quốc gia ở châu Âu” (How many countries in Europe) forms 5 words “Có bao_nhiêu How many quốc_gia country ở in châu_Âu Europe ”.
Thus it is interesting to study the influence of word segmentation in Vietnamese on its SQL parsing, i.e. syllable level vs. word level.
In terms of Vietnamese semantic parsing, previous approaches construct rule templates to convert single database-driven questions into meaning representations (Nguyen and Le, 2008; Nguyen et al., 2009, 2012;
Tung et al., 2015; Nguyen et al., 2017).
Recently, Vuong et al.
(2019) formulate the Textto-SQL semantic parsing task for Vietnamese as a sequence labeling-based slot filling problem, and then solve it by using a conventional CRF model with handcrafted features, due to the simple structure of the input questions they deal with.
Note that seq2seq-based semantic parsers have not yet been explored in any previous work w.r.t.
Vietnamese.
Semantic parsing datasets for Vietnamese include a corpus of 5460 sentences for assigning semantic roles (Phuong et al., 2017) and a small Textto-SQL dataset of 1258 simple structured questions over 3 databases (Vuong et al., 2019).
However, these two datasets are not publicly available for research community.
In this paper, we introduce the first public largescale Text-to-SQL dataset for the Vietnamese semantic parsing task.
In particular, we create this dataset by manually translating the Spider dataset into Vietnamese.
We empirically evaluate strong seq2seq baseline parsers EditSQL (Zhang et al., 2019) and IRNet (Guo et al., 2019) on our dataset.
Extending the baselines, we extensively investigate key configurations and find that: (1) Our human-translated dataset is far more reliable than a dataset consisting of machine-translated questions, and the overall result obtained for Vietnamese is comparable to that for English.
(2) Automatic Vietnamese word segmentation improves the performances of the baselines.
The NPMI score (Bouma, 2009) is useful for linking a cell value mentioned in a question to a column in the database schema.
(4) Latent syntactic features, which are dumped from a neural dependency parser pre-trained for Vietnamese (Nguyen and Verspoor, 2018), also help improve the performances.
Highest improvements are accounted for the use of pre-trained language models, where PhoBERT (Nguyen and Nguyen, 2020) helps produce higher results than XLM-R (Conneau et al., 2020).
We hope that our dataset can serve as a starting point for future Vietnamese semantic parsing research and applications.
We publicly release our dataset at: https://github.com/ VinAIResearch/ViText2SQL .
Our Dataset
We manually translate all English questions and the database schema (i.e. table and column names as well as values in SQL queries) in Spider into Vietnamese.
Note that the original Spider dataset consists of 10181 questions with their corresponding 5693 SQL queries over 200 databases.
However, only 9691 questions and their corresponding SQL queries over 166 databases, which are used for training and development, are publicly available.
Thus we could only translate those available ones.
The translation work is performed by 1 NLP researcher and 2 computer science students (IELTS [Table]
“#Qu.”, “#SQL” and “#DB” denote the numbers of questions, SQL queries and databases, respectively. “#T/D” abbreviates the average number of tables per database. “#Easy”, “#Med.”, “#Hard” and “#ExH” denote the numbers of questions categorized by their SQL queries’ hardness levels of “easy”, “medium”, “hard” and “extra hard”, respectively (as defined by Yu et al.).
7.0+).
Every question and SQL query pair from the same database is first translated by one student and then cross-checked and corrected by the second student; and finally the NLP researcher verifies the original and corrected versions and makes further revisions if needed.
Note that in case we have literal translation for a question, we stick to the style of the original English question as much as possible.
Otherwise, for complex questions, we will rephrase them based on the semantic meaning of the corresponding SQL queries to obtain the most natural language questions in Vietnamese.
Following Yu et al.
(2018) and Min et al.
(2019), we split our dataset into training, development and test sets such that no database overlaps between them, as detailed in Table 1.
Examples of question and SQL query pairs from our dataset are presented in Table 2.
Note that translated question and SQL query pairs in our dataset are written at the syllable level.
To obtain a word-level version of the dataset [Table] we apply RDRSegmenter (Nguyen et al., 2018) from VnCoreNLP (Vu et al., 2018) to perform automatic Vietnamese word segmentation
[Table]
3. Baselines
Recent state-of-the-art results on the Spider dataset are reported for RYANSQL (Choi et al., 2020) and
RAT-SQL (Wang et al., 2020), which are based on the seq2seq encoder-decoder architectures.
However, their implementations are not published at the time of our empirical investigation.
Thus we select seq2seq based models EditSQL (Zhang et al., 2019) and IRNet (Guo et al., 2019) with publicly available implementations as our baselines, which produce near state-of-the-art scores on Spider. We briefly describe the baselines EditSQL and IRNet as follows:
• EditSQL is developed for a context-dependent Text-to-SQL parsing task, consisting of: (1) a BiLSTM-based question-table encoder to explicitly encode the question and table schema, (2) a BiLSTM-based interaction encoder with attention to incorporate the recent question history, and (3) a LSTM-based table-aware decoder with attention, taking into account the outputs of both encoders to generate a SQL query. • IRNet first performs an n-gram matching-based schema linking to identify the columns and the tables mentioned in a question. Then it takes the question, a database schema and the schema linking results as input to synthesize a tree-structured SemQL query—an intermediate representation bridging the input question and a target SQL query. This synthesizing process is performed by using a BiLSTM-based question encoder and an attention-based schema encoder together with a grammar-based LSTM decoder (Yin and Neubig, 2017).
Finally, IRNet deterministically uses the synthesized SemQL query to infer the SQL query with domain knowledge.
See Zhang et al.
(2019) and Guo et al.
for more details of EditSQL and IRNet, respectively.
3. Our Extensions
NPMI for schema linking: IRNet essentially relies on the large-scale knowledge graph ConceptNet (Speer et al., 2017) to link a cell value mentioned in a question to a column in the database schema, based on two ConceptNet categories ‘is a type of’ and ‘related terms’.
However, these two ConceptNet categories are not available for Vietnamese.
Thus we propose a novel use of the NPMI collocation score (Bouma, 2009) for the schema linking in
IRNet, which ranks the NPMI scores between the cell values and column names to match a cell value to its column.
features: Previous works have shown that syntactic features help improve semantic parsing (Monroe and Wang, 2014;
Jie and Lu, 2018)
Unlike these works that use handcrafted syntactic features extracted from dependency parse trees, and inspired by Zhang et al.
(2017)’s relation extraction work, we investigate whether latent syntactic features, extracted from the BiLSTM-based dependency parser jPTDP (Nguyen and Verspoor, 2018) pre-trained for Vietnamese, would help improve Vietnamese Text-to-SQL parsing.
In particular, our approach is that we dump latent feature representations from jPTDP ’s BiLSTM encoder given our word-level inputs, and directly use them as part of input embeddings of EditSQL and IRNet.
Pre-trained language models: Zhang et al.
(2019) and Guo et al.
(2019) make use of BERT (Devlin et al., 2019) to improve their model performances.
Thus we also extend EditSQL and IRNet with the use of pre-trained language models XLMR-base (Conneau et al., 2020) and PhoBERT-base (Nguyen and Nguyen, 2020) for the syllable- and word-level settings, respectively.
XLM-R is the recent best multi-lingual model, based on RoBERTa (Liu et al., 2019), pre-trained on a 2.5TB multilingual corpus which contains 137GB of syllable-level
Vietnamese texts.
PhoBERT is a monolingual variant of RoBERTa for Vietnamese, pre-trained on a 20GB of word-level Vietnamese texts.
Experiments 4. Experimental Setup
We conduct experiments to study a quantitative comparison between our human-translated dataset and a machine-translated dataset, 2 the influence of
Vietnamese word segmentation (i.e. syllable level and word level), and the usefulness of the latent syntactic features, the pre-trained language models and the NPMI-based approach for schema linking. For both baselines EditSQL and IRNet which require input pre-trained embeddings for syllables Approach BY
[Table]
[Table]
and words, we pre-train a set of 300-dimensional syllable embeddings and another set of 300dimensional word embeddings using the Word2Vec skip gram model (Mikolov et al., 2013) on syllableand word-level corpora of 20GB Vietnamese texts (Nguyen and Nguyen, 2020). In addition, we also use these 20GB syllable- and word-level Vietnamese corpora as our external datasets to compute the NPMI score (with a window size of 20) for schema linking in IRNet. Our hyperparameters for EditSQL and IRNet are taken from Zhang et al. and Guo et al.
(2019), respectively.
The pre-trained syllable and word embeddings are fixed, while the pre-trained language models XLM-R and PhoBERT are finetuned during training. Following Yu et al. (2018), we use two commonly used metrics for evaluation. The first one is the exact matching accuracy, which reports the percentage of input questions that have exactly the same SQL output as its gold reference.
The second one is the component matching F 1 , which reports F 1 scores for SELECT, WHERE, ORDER
BY,
GROUP
BY and all other keywords.
We run for 10 training epochs and evaluate the exact matching accuracy after each epoch on the development set, and then select the best model checkpoint to report the final result on the test set.
4. Main Results
Table 3 shows the overall exact matching results of EditSQL and IRNet on the development and test sets.
Clearly, IRNet does better than EditSQL, which is consistent with results obtained on the original English Spider dataset.
We find that our human-translated dataset is far more reliable than a dataset consisting of machinetranslated questions.
In particular, at the word level, compared to the machine-translated dataset, our dataset obtains about 30.2-17.4 ≈ 13% and 43.621.6 = 22% absolute improvements in accuracies of
EditSQL and IRNet, respectively (i.e. 75%–100% relative improvements).
In addition, the word-based
Text-to-SQL parsing obtains about 5+% absolute higher accuracies than the syllable-based Text-toSQL parsing (EditSQL: 24.1%
→
30.2% ; IRNet: 38.2% → 43.6%), i.e. automatic Vietnamese word segmentation improves the accuracy results.
Furthermore, latent syntactic features dumped from the pre-trained dependency parser jPTDP for Vietnamese help improve the performances of the baselines (EditSQL: 30.2% → 42.2%; IRNet: 43.6% → 47.1%).
Also, biggest improvements are accounted for the use of pre-trained language models.
In particular, PhoBERT helps produce higher results than XLM-R (EditSQL: 52.6% vs. 51.3%;
IRNet: 53.2% vs. 52.8%).
We also retrain EditSQL and IRNet on the English Spider dataset with the use of the strong pretrained language model RoBERTa instead of BERT, but under our dataset split.
We find that the overall results for Vietnamese are smaller but comparable to the English results.
Therefore, Text-to-SQL semantic parsing for Vietnamese might not be significantly more challenging than that for English.
Table 4 shows the exact matching accuracies of
EditSQL and IRNet w.r.t.
different hardness levels of SQL queries and the F 1 scores w.r.t.
different SQL components on the test set.
Clearly, in most cases, the pre-trained language models PhoBERT and XLM-R help produce substantially higher results than the latent syntactic features, especially for the WHERE component.
NPMI-based schema linking: We also investigate the contribution of our NPMI-based extension approach for schema linking in applying IRNet for Vietnamese.
Without using NPMI for schema linking, 3 we observe 6+% absolute decrease in the exact matching accuracies of IRNet on both development and test sets, thus showing the usefulness of our NPMI-based approach for schema linking.
4. Error Analysis
To understand the source of errors, we perform an error analysis on the development set which consists of 954 questions.
Using IRNet PhoBERT which produces the best result, we identify several causes of errors from 382/954 failed examples.
For 121/382 cases (32%), IRNet PhoBERT makes incorrect predictions on the column names which are not mentioned or only partially mentioned in the questions.
For example, given the question “Hiển thị tên và năm phát hành của những bài hát thuộc về ca sĩ trẻ tuổi nhất” (Show the name and the release year of the song by the youngest singer), 4 the model produces an incorrect column name prediction of “tên” (name) instead of the correct one “tên bài hát” (song name).
Errors related to column name predictions can either be missing the entire column names or inserting random column names into the
WHERE component of the predicted SQL queries.
About 12% of failed examples (47/382) in fact have an equivalent implementation of their intent with a different SQL syntax.
For example, the model produces a ‘failed’ SQL output “ SELECT
MAX [sức chứa] FROM [sân vận động]” which is equivalent to the gold SQL query of “ SELECT [sức chứa] FROM [sân vận động] ORDER BY [sức chứa]
DESC LIMIT 1 ”, i.e. the SQL output would be valid if we measure an execution accuracy.
About 22% of failed examples (84/382) are caused by nested and complex SQL queries which mostly belong to the Extra Hard category.
With 18% of failed examples (70/382), incorrectly predicting operators is another common type of errors.
For example, given the phrases “ già nhất ” (oldest) and “ trẻ nhất ” (youngest) in the question, the model fails to predict the correct operators max and min , respectively.
The remaining 60/382 cases (16%) are accounted for an incorrect prediction of table names in a FROM clause.
Conclusion
In this paper, we have presented the first public large-scale dataset for Vietnamese Text-to-SQL semantic parsing.
We also extensively experiment with key research configurations using two strong baseline models on our dataset and find that: the input representations, the NPMI-based approach for schema linking, the latent syntactic features and the pre-trained language models all have the influence on this Vietnamese-specific task.
We hope that our dataset can serve as the starting point for further research and applications in Vietnamese question answering and dialogue systems.