S ecure SQL : Evaluating Data Leakage of Large Language Models as Natural Language Interfaces to Databases (2024)

Abstract: With the widespread application of Large Language Models (LLMs) in Natural Language Interfaces to Databases (NLIDBs), concerns about security issues in NLIDBs have been increasing gradually. However, research on sensitive data leakage in NLIDBs is relatively limited. Therefore, we propose a benchmark to assess the potential of language models to leak sensitive data when generating SQL queries. This benchmark covers 932 samples from 34 different domains, including medical, legal, financial, and political aspects. We evaluate 15 models from six LLM families, and the results show that the model with the best performance has an accuracy of 61.7%, whereas humans achieve an accuracy of 94%. Most models perform close to or even below the level of random selection. We also evaluate two common attack methods, namely prompt injection and inference attacks, as well as a defense method based on chain-of-thoughts (COT) prompting. Experimental results show that both attack methods significantly impact the model, while the defense method based on COT prompting dose not significantly improve accuracy, further highlighting the severity of sensitive data leakage issues in NLIDBs. We hope this research will draw more attention and further study from the researchers on this issue.

## introduction text-to-sql, also known as natural language interface for databases (nlidbs), is a technique that converts natural language questions from users into sql queries. this approach reduces the technical barrier to using databases while promoting widespread data utilization and intelligent applications. in today's data-driven environment, this task is of considerable importance. ## discussion the poor performance of baselines in se-curesql highlights the urgency of addressing llm data leakage issues in nlidbs. the baseline models perform poorly in securesql, showing a significant gap from human average levels. their weak performance in inference attacks and adversarial samples highlights the challenge of understanding natural language semantics and sql query intent. various types of prompt injection notably degrade model performance, indicating that this type of attack is serious and cannot be overlooked. more concerning is that the hybrid attack can render the model completely defenseless, further emphasizing the urgency of addressing data leakage issues in nlidbs. an auxiliary llm guardian based on cot has proven to be a viable defense solution. it enhances the model's reasoning ability, making it particularly effective against inference attacks. how-ever, for attacks that do not rely entirely on reasoning, such as direct and prior-based attacks, this approach may inadvertently lead to negative impacts, similar to the findings by @xcite . therefore, exploring new paradigms that can defend against various types of attacks remains a valuable area for further research. the conflicting performance between recall and specificity also poses a significant challenge in this task. securesql is designed to balance defense capabilities with the ability to avoid misjudgments, making it highly valuable for evaluating model performance in real-world applications. ## conclusion in this paper, we explore the issue of sensitive data leakage in natural language interfaces to databases (nlidbs) and propose a benchmark called securesql. we conduct experiments on 15 popular large language models (llms), and the results show that this issue are widespread in these models. additionally, we also study relevant attack and defense methods. we find that nlidbs based on llms are susceptible to inference attacks and prompt injection attacks. however, the effectiveness of defense methods based on chain-of-thoughts (cot) prompting is limited, further highlighting the seriousness of sensitive data leakage in nlidbs. we hope this work will inspire researchers to further investigate the issue of sensitive data protection on nlidbs.