PAUQ : Text-to- SQL in R ussian (2022)

Abstract: Semantic parsing is an important task that allows to democratize human-computer interaction. One of the most popular text-to-SQL datasets with complex and diverse natural language (NL) questions and SQL queries is Spider. We construct and complement a Spider dataset for Russian, thus creating the first publicly available text-to-SQL dataset for this language. While examining its components - NL questions, SQL queries and databases content - we identify limitations of the existing database structure, fill out missing values for tables and add new requests for underrepresented categories. We select thirty functional test sets with different features that can be used for the evaluation of neural models’ abilities. To conduct the experiments, we adapt baseline architectures RAT-SQL and BRIDGE and provide in-depth query component analysis. On the target language, both models demonstrate strong results with monolingual training and improved accuracy in multilingual scenario. In this paper, we also study trade-offs between machine-translated and manually-created NL queries. At present, Russian text-to-SQL is lacking in datasets as well as trained models, and we view this work as an important step towards filling this gap.

## introduction semantic parsing (sp) is the task of transforming a natural language (nl) utterance to a formal machine-understandable representation. such representations include a variety of forms: from parsing linguistic features to generating a code in a specific programming language. in this paper, we focus on a sp subtask -mapping of nl questions to structured query language (text-to-sql). machine learning systems which are based on supervised learning and aimed at solving text-to-sql, continue to appear and evolve actively @xcite @xcite . the key ingredient in the process of training is the data: a parallel corpus of nl questions and corresponding sql queries along with the databases. the majority of text-to-sql datasets consist of questions and database contents written in english. as a consequence, the development of such models has been limited to the english language. spider dataset @xcite remains one of the most popular benchmarks in the field due to the variety of domains and complexity of the questions. this paper presents the russian version of the spider: pauq foot_0 , the first text-to-sql dataset in russian foot_1 . in pauq, all three components have been modified and localized: the nl questions, the sql queries, and the content of the databases. during this in-depth work, we discover several limitations of the original spider and propose ways to overcome them. the new version of spider database collection is presented. apart from being bilingual, it is more complete -in order to use not only exact match as the evaluation metric but also execution accuracy, avoiding falsepositive results. we also complement the dataset with the new samples of underrepresented types, including questions regarding columns with binary values, columns containing date and time values, and the ones that have a fuzzy and partial match with the database content. we adapt and evaluate on pauq two strong ml models relating to different types of architecture: rat-sql (grammar-based) and bridge (sequence-to-sequence). we compare the performance of these models in terms of component understanding of sql and schema linking. our evaluation demonstrates that both models present strong results with monolingual training and improved accuracy in a multilingual set-up. another important result of this work is the development of functional test sets, subsets of samples in english and russian with particular features. they can be used by other researchers to evaluate the models' performance on particular types of questions -thus, in a more precise and informative way. our experiments with machinetranslated data suggest that in terms of resources expended and model performance, it could be beneficial to use machine translation for easy cases (such as questions not containing values) and manual translation for questions that imply referring to the database values. we regard this work as an important step in russian text-to-sql. furthermore, we hope that our contributions will help other researchers in adaptation of spider to other languages. ## related work text-to-sql is a topic that has been actively studied in recent years with a number of datasets and benchmarks @xcite @xcite @xcite . the goal of the spider benchmark is to develop nl interfaces to cross-domain databases. it consists of 10,181 questions and 5,693 unique complex sql queries on 200 databases with multiple tables covering 138 different domains. in spider 1.0, different complex sql queries and databases appear in train and test sets. to do well on it, systems must generalize well to not only new sql queries but also new database schemas. there have been several attempts to adapt the spider dataset to other languages. the first one is chinese spider (cspider) @xcite with original questions from the spider dataset translated from english into chinese. in this case, the translation is not literal: a specific cultural localization of some values was conducted. the principles of this localization, however, were not explicitly stated. authors also compare results obtained by the models trained on the machine-translated questions with the results of the models that were trained on nl queries translated manually and subjected to localization. the study shows that models trained on human translation significantly outperform machine-translation-based. this fact highlights the idea that adaptation of english datasets to other languages requires professional translation and localization. however, databases content in cspider was not complemented by new localized values, so its impossible to make a complete evaluation of the obtained results. another two specific versions of the spider dataset are vietnamese @xcite and @xcite . in the case of vietnamese, not only nl questions were translated, but also the database schema, including table and column names along with values in sql queries. translated values, unlike those of chinese spider, were not localized; they also were not added to the databases content. in line with the experimental setup of research on cspider, a comparison between manually-translated and machine-translated versions of the vietnamese dataset was conducted. in portuguese spider only nl questions, excluding database values, were translated, thus no modifications affecting the database schema or content were made. as can be seen, in all currently existing versions of the spider dataset, although they differ in principles by which the language adaptation is carried out, not all components of the original dataset are modified: database content is left unchanged and isn't aligned with the questions. it also means that execution accuracy can't be used to measure models' performance, thus only exact match is reported. ## dataset when managing the translation process of the original spider dataset into russian, we were guided by a similar experience with the translation into chinese @xcite . after careful review of the chinese spider (cspider), we identified two aspects that were not reflected in it -and which we would like to take into account: the formulation of the principles according to which the original english query has to be modified during translation in order to comply with russian language and culture; the insertion of the modified values into database content so that models could be evaluated with the execution accuracy correctly. as database content is enriched by russian values, we should pay regard to another significant issue -unambiguity of the questions. this principle also means that russian values shouldn't be direct translations of english values as in this case it's not obvious what results we expect to get: should sql queries be formulated so that english values are taken into consideration or should it be rus-sian values or, perhaps, a combination of both languages. for real-world databases in russia, just like in brazil, according to (josé and cozman, 2021), it is common to have english table and column names although the content can be in russian. for this reason, all table and column names in sql queries and database schemas remain in english. the main principle of translation is the following: if there are only tables/columns names in an sql query and no values, such query and corresponding text question are not subjected to localization or modification -they are just translated; in case an sql query contains values, this query as well as corresponding text question are changed on the basis of several criteria foremost among which is the following: russian values should not be literal translations of english ones, but rather should be close analogues that are at the same time absent in the original database content. principles of values localization could be find in appendix a. the translation from english to russian is undertaken by a professional human translator. the original spider dataset consists of 10,181 samples, however the test set is closed which means that only 9,691 of them are available for processing. preliminary work includes dividing questions into those that have values -thus, need modifications -and those that do not. for questions with values, lists with all corresponding column's values from database are obtained in order to ease the selection process of russian values that do not have english direct analogues in the database. the translator is in advance informed of the sql query structure and provided with the instruction in which all basic principles of the translation are stated. the translator is encouraged to translate question as closely to the original text as possible and rephrase it as necessary to obtain most natural russian text. we also strive for a variety of questions, that is why no textual patterns are used. as regards the updating of the database content, we added all localized/modified values to the corresponding fields. without this, fair evaluation of the models with the execution accuracy would be impossible as for the questions containing absent values, the ground truth and predicted requestsincorrect as well as correct -would return the same result when executed. in case of binary values, we delete english analogues in order to avoid ambiguity and keep the column's values binary. a verification of the translated questions and their conformity with the queries, and an updating of the databases are undertaken by 4 computer science students. at first, all databases with corresponding questions in russian and sql queries are divided between annotators. each of the students checks the correctness of a question and a query; if the values in the query has been changed (there is also a special mark indicating it), the annotator adds the new values to the corresponding fields of the database table(s) so that when the database is accessed by the query, the added values are retrieved. after that the very same question and query are cross-checked by another annotator who also makes sure that the database request works as expected. we made a revision of the original spider dataset: for those sql queries that return none or empty list, corresponding values are added to the original databases; databases that were empty (geo, scholar, yelp, academic, imdb, restaurants) are filled with values. further information on these changes could be find in subsection 4.1. following research that has been conducted previously on building the spider dataset for other languages, we also constructed machinetranslated (mt) dataset, using yandex translate api @xcite . the experiments with chinese and vietnamese spider versions show that the performance of the models trained on such data drops compared to that of the models trained on human-translated dataset. the quality of mt dataset certainly depends on the quality of the machine translation model itself; in addition, there is even more problematic issue -the translation of values. without human revision and database updating, values in nl questions will not match database content in most cases. in order to examine the trade-off between the performance of the models trained on the data and the resources spent to obtain this data, we also experiment with the combined dataset, in which questions without values are machine-translated whereas for questions that include values, manual translation is used to avoid inconsistency (see subsection 5.2). ## comparative analysis besides manual translation of the questions from the original spider we (i) add new samples, (ii) insert in the tables new values, and (iii) compose a suit of functional test sets. in order to assess whether the basic properties of the spider are not violated and whether its limitations are improved, we make a detailed analysis of the datasets and present it in the appendix b. here we outline the motivation for the updates, highlight the main differences and define an approach for estimation of the key dataset features like complexity, balance and diversity. ## experiments in this paper, we provide an empirical evaluation of baseline neural network models for text-to-sql on our dataset. to date, there are many different models present and there are three major categories of these models @xcite : (i) sequence-to-sequence, (ii) grammar-based, and (iii) sketch-based slot filling. we have analyzed the spider leaderboard @xcite and concluded that the sketch-based slot filling method does not stand up to the competition with the other two methods, which leaves us with two other approaches. we kept in mind the following requirements for models during our selection: according to our requirements, we utilized two popular spider models -rat-sql (grammarbased) @xcite ) and bridge (sequence-to-sequence) (lin et al. 2020). rat-sql is an encoder-decoder framework that uses relation-aware transformer within the encoder to model alignments between database schema and content and question tokens. the decoder of the model is tree-structured and generates abstract syntax tree in the context-free sql grammar. bridge, in turn, utilizes database schema and content as input to the model. it has an encoderdecoder architecture with the pointer-generator network using beam-search. the model generates queries in execution-guided order. within the experiments, we seek to answer to following research questions (rqs): rq1 what is the performance difference between models which were trained on english, or on various russian datasets, or in multilingual setup (english and russian)? rq2 do we need to use qualified human translation while adapting the original dataset to different language (in our case, russian)? rq3 how does query component prediction differ depending on the language? ## conclusion in this paper, we have presented the first public text-to-sql dataset for the russian language. based on the spider -large-scale, cross-domain benchmark composed of table interconnected databases and corresponding nl queries. we improved the original spider by inserting the missing values, correcting errors, and adding new samples of poorly represented types. to answer the stated research questions, we conducted experiments with rat-sql and bridge architectures trained on different combinations of russian and english data. based on the results obtained, we got the following key observations. lexical and semantic diversity of questions, queries and table values was improved. extracted categories of requests with different artifacts in mentioned entities, question formulation and logical structure show model weaknesses and open perspectives for further development. although russian and english differ considerably from each other, similar models show similar performance on these languages. as we can see, errors in the structure and logic components are quite similar, but errors related to schema linking are different: it is much harder for russian models to generalize and match on these components than for english ones. machine translation works in comparably for queries without values for selected text-to-sql models, but human translation is needed for queries with values in order to get qualitative data and provide better results. our experiments show that training on russian and english languages simultaneously using a multilingual encoder brings a notable increase on both russian and english development datasets. we mark the development of multilingual models as a prospect for future work.