W av2 SQL : Direct Generalizable Speech-To- SQL Parsing (2024)

Abstract: We release a multi-accent dataset and propose speech-programming and gradient reversal classifier to improve the generalization.Abstract: Speech-to-SQL (S2SQL) aims to convert spoken questions into SQL queries given relational databases, which has been traditionally implemented in a cascaded manner while facing the following challenges: 1) model training is faced with the major issue of data scarcity, where limited parallel data is available; and 2) the systems should be robust enough to handle diverse out-of-domain speech samples that differ from the source data. In this work, we propose the direct generalizable speech-to-SQL parsing model Wav2SQL which avoids error compounding across cascaded systems. Specifically, 1) to accelerate speech-driven SQL parsing research in the community, we release a large-scale and multi-accent dataset MASpider; 2) leveraging the recent progress in the large-scale pre-training, we show that it alleviates the data scarcity issue and allow for direct speech-to-SQL parsing; and 3) we include the speech re-programming and gradient reversal classifier techniques to reduce acoustic variance and learned style-agnostic representation, improving generalization to unseen out-of-domain custom data. Experimental results demonstrate that Wav2SQL avoids error compounding and achieves state-of-the-art results by up to 4.7% accuracy improvement over the baseline.

## introduction speech-to-sql parsing (s2sql) aims to generate the sql query from a spoken question based on relational databases. this technology is highly beneficial as it breaks down barriers among those who lack proficiency in sql queries and are unable to perform screen inputs while driving or exercising. furthermore, s2sql provides flexible and convenient ways of interaction, which opens up a host of practical applications in fields such as vehicle terminals, smart watches, smart speakers, and the medical industry. conventional s2sql systems @xcite are often composed of a cascade of two components: automatic speech recognition (asr) @xcite @xcite ) and text-to-sql parsing @xcite @xcite . compared to cascaded systems, work on direct s2sql is very limited, with the potential benefits of 1) working on languages without written form @xcite , where an estimated half of the 7,000 languages in the world actually do not have written forms; 2) avoiding error compounding across sub-systems @xcite . the recent development of direct s2sql parsing still faces several challenges: 1) despite the benefits of direct approaches, model training is faced with the major issue of data scarcity. humanlabeled speech data is expensive to create, there are very few data resources providing parallel speech, and the data amount is quite limited, 2) increasing demand for sql parsing from personalized speech challenges models especially in unseen scenarios. when the distributions of custom voice (speaker and accent) differ from training data, the system performance deteriorates due to distribution gaps, and 3) the modality gap between the spoken question and text schema hinders the ability of the scheme linking, making it difficult to align question speech to the intended tables. to accelerate s2sql research, we assemble an open-source, multi-speaker, and multi-accent s2sql corpus maspider. to the best of our knowledge, maspider is the first open-source speechto-sql parsing dataset. we have attached part of maspider to the supplementary materials, and we will release the entire dataset after the paper publication. to overcome the aforementioned challenges in this paper, we propose wav2sql for direct speech-to-sql parsing, which is generalizable to unseen acoustic conditions (speaker and accent) in custom data. to be more specific, 1) leveraging self-supervised learning (ssl) @xcite , it alleviates the data scarcity issue and benefits s2sql model training, 2) we introduce speech re-programming and gradient reverse technique to effectively eliminate the style attributes in representation, which promote the model generalization to unseen speakers and accents in custom data. experimental results on the maspider dataset demonstrate that our wav2sql model surpasses the cascaded system in the exact match accuracy and achieves competitive performance with our model trained on the tts dataset. the main contributions of this work are summarized as follows: ## dataset construction we build maspider upon the spider @xcite , which has 8659/1034 train/evaluation splits and an unreleased test set. maspider consists of 9693 spoken utterances recorded by eleven speakers from six different countries. maspider consists of 15 hours of speech samples recorded in a professional recording studio, including 8.1 hours from 6 females and 6.9 hours from 5 males apart from the person-of-interest (poi). figures 1 summarize the distribution of dataset split, gender, and country. more details on maspider are available in the appendix d. the major features of maspider include: ## conclusion we released maspider, the first large-scale, multispeaker, and multi-accent s2sql parsing dataset, which we hope would accelerate s2sql research in the community. in this work, we presented the first direct speech-to-sql model wav2sql which avoided error compounding across cascaded systems. to tackle the data scarcity issue, we leveraged recent progress in large-scale pre-training and utilized self-supervised models to derive discriminate representation. to promote model generalization and robustness to custom out-of-distribution data, we further introduced speech re-programming and gradient-reversal classifier techniques which reduced acoustic variance and learned style-agnostic representations. experimental results demonstrated that our approach achieved new state-of-the-art results by up to 4.7% accuracy improvement over baseline. in the future, we will investigate techniques to further enhance the model generalization in direct speech-to-sql parsing.