Federated Learning for Semantic Parsing: Task Formulation, Evaluation Setup, New Algorithms (2023)

Abstract: This paper studies a new task of federated learning (FL) for semantic parsing, where multiple clients collaboratively train one global model without sharing their semantic parsing data. By leveraging data from multiple clients, the FL paradigm can be especially beneficial for clients that have little training data to develop a data-hungry neural semantic parser on their own. We propose an evaluation setup to study this task, where we re-purpose widely-used single-domain text-to-SQL datasets as clients to form a realistic heterogeneous FL setting and collaboratively train a global model. As standard FL algorithms suffer from the high client heterogeneity in our realistic setup, we further propose a novel LOss Reduction Adjusted Re-weighting (Lorar) mechanism, which adjusts each client’s contribution to the global model update based on its training loss reduction during each round. Our intuition is that the larger the loss reduction, the further away the current global model is from the client’s local optimum, and the larger weight the client should get. By applying Lorar to three widely adopted FL algorithms (FedAvg, FedOPT and FedProx), we observe that their performance can be improved substantially on average (4%-20% absolute gain under MacroAvg) and that clients with smaller datasets enjoy larger performance gains. In addition, the global model converges faster for almost all the clients.

## introduction semantic parsing aims to translate natural language utterances into formal meaning representations such as sql queries and api calls and can be applied to build natural language interfaces that enable users to query data and invoke services without programming @xcite @xcite . neural semantic parsers have achieved remarkable performance in recent years @xcite @xcite . however, they are data-hungry; bootstrapping a neural semantic parser by annotating data on a large scale can be very challenging for many institutions, as it requires the annotators to have intimate knowledge of formal programs. one natural thought is to leverage data from different institutions and train a unified model that can be used for all institutions. however, in practice, institutions such as hospitals, banks, and legal firms are prohibited from sharing their data with others, due to privacy concerns. therefore, for institutions that only have very limited data, it is extremely hard to build their own neural semantic parsers. federated learning (fl) @xcite @xcite has turned out to be a popular training paradigm where multiple clients can collaboratively train a global model without exchanging their own data. in this paper, we study a new task of federated learning for semantic parsing. through fl on the data scattered on different clients (e.g., institutions), we aim to obtain a global model that works well for all clients, especially those that have insufficient data to build their own neural models. towards that end, we propose an evaluation setup by re-purposing eight existing datasets that are widely adopted for text-to-sql parsing, such as atis @xcite and @xcite . these datasets demonstrate great heterogeneity, in terms of dataset sizes, language usage, database structures, and sql complexity, as they were collected from the real life by different researchers, at different times, and for different purposes. therefore, we use this collection to simulate a realistic scenario where eight clients with very different data participate in the fl paradigm to jointly train a neural semantic parser. heterogeneity, where the data distributions and dataset sizes on different clients are different, is recognized as one of the biggest challenges in fl @xcite @xcite @xcite . existing work either uses synthetic data @xcite or splits a classification dataset based on dirichlet distribution @xcite to simulate the non-iid federated learning setting, while we propose a more realistic setup to study this setting for semantic parsing. pre-trained language models such as t5 @xcite have been shown as a powerful unified model for various semantic parsing tasks @xcite , which can be leveraged to save us the efforts for client-specific model designs. specifically, we adopt t5-base as our backbone semantic parser in the fl paradigm, and conduct extensive experiments and analysis using three widelyadopted fl algorithms: fedavg @xcite , fedopt @xcite and fedprox @xcite . as standard fl algorithms suffer from the high client heterogeneity in our realistic setup, we further propose a novel re-weighting mechanism for combining the gradient updates from each client during the global model update. the high-level idea is shown in figure 1 . our intuition is that, for each client, the reduction of training loss during each round can signalize how far the current global model is away from the local optimum. by giving larger weights to those clients that have larger training loss reduction, the global model update can accommodate those clients better, thus mitigating potential performance degradation caused by high heterogeneity. we formulate this intuition as a re-weighting factor to adjust how much each client should contribute to the global model update during each round. our proposed mechanism can be applied to all the three fl algorithms and experiments show that it can substantially improve both their parsing performance and their convergence speed, despite being very simple. in summary, our main contributions are: ## evaluation setup as we are the first to study cross-silo fl for semantic parsing, there is no benchmark for this task. thus we establish an evaluation setup by re-purposing eight single-domain text-to-sql datasets @xcite as eight "clients", which demonstrate high heterogeneity in terms of dataset sizes, domains, language usage, database structures and sql complexity. given a natural language question and the database schema, text-to-sql parsing aims to generate a sql query. here the question is a sequence of tokens and the database schema consists of multiple tables with each table containing multiple columns. figure 7 in appendix shows an example of this task. we adopt t5-base as our backbone model, which has been shown as an effective unified model for various semantic parsing tasks @xcite . similarly as in previous work @xcite , we concatenate the question tokens with the serialized relational table schemas (table names and column names) as the model input and output a sequence of sql tokens. the heterogeneity of the eight clients is described in detail from the following perspectives. domain: the clients are from diverse domains. some clients such as scholar and academic are from closer domains than others. ## fl for semantic parsing in this section, we first introduce the background of fl, more specifically, its training objective, training procedure and three widely adopted fl algorithms. then we describe the motivating insights and details of our proposed mechanism. ## experiments datasets. we re-purpose eight datasets: atis @xcite , geoquery @xcite , @xcite @xcite , scholar (srinivasan iyer and zettlemoyer, 2017), academic @xcite , advising @xcite , yelp and imdb (navid yaghmazadeh and dillig, 2017) as eight clients. these datasets have been standardized to the same sql style by @xcite . their characteristics have been described in section 3. we follow "question split" datasets preprocessed by @xcite to split the train, dev and test data, which means we let the train, dev and test examples have different questions but the same sql queries are allowed. for advising, atis, geo-query and scholar, we directly use the original question split as our split. for restaurants, academic, imdb and yelp, since the data sizes are relatively small, the original question split uses 10 splits for cross validation without specifying train, dev and test examples. given fl is costly as we need multiple gpus to finish one experiment, we fix the train, dev and test set by randomly selecting 6 splits as the train set, 2 splits as the dev set and 2 splits as the test set. evaluation metrics. 1) exact match (em): a prediction is deemed correct only if it is exactly the same as the ground truth (i.e., exact string match), which is widely used for text-to-sql parsing @xcite . all the evaluations in our experiments consider the values generated in the sql query. 2) macroavg: the arithmetic mean of em across all clients, which treats each client equally. 3) microavg: the total number of correct predictions on all the clients divided by the total test examples, which treats each test example equally. learning paradigm. we compare three learning paradigms: finetuning, centralized and fl. 1) finetuning: we individually finetune our backbone model (t5-base) on the training data of each client. for the computing resources, we use 1 nvidia a6000 48gb gpu for finetuning, with batch size 8. we use 2 nvidia a6000 48gb gpus for central-ized training, with batch size 8. we use 5 nvidia a6000 48gb gpus for all federated learning experiments. specifically, one gpu is used as the server and the other four gpus are used as 8 clients, with each gpu accommodating 2 clients. the batch size for clients geoquery, restaurants, scholar, academic, imdb and yelp is 4, and for clients advising and atis is 8. ## conclusions to the best of our knowledge, we are the first to study federated learning for semantic parsing. specifically, we propose a realistic benchmark by re-purposing eight single-domain text-to-sql datasets. moreover, we propose a novel loss reduction adjusted re-weighting mechanism (lorar) that is applicable to widely adopted fl algorithms. by applying lorar to fedavg, fedopt and fed-prox, we observe their performance can be improved substantially on average, and clients with smaller datasets enjoy larger performance gains.