richardr1126 commited on
Commit
2e85252
1 Parent(s): 32f3acc

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +12 -0
README.md CHANGED
@@ -26,6 +26,18 @@ The goal of the Spider challenge is to develop natural language interfaces to cr
26
 
27
  This dataset was used to finetune this model.
28
 
 
 
 
 
 
 
 
 
 
 
 
 
29
  ## Citation
30
 
31
  Please cite the repo if you use the data or code in this repo.
 
26
 
27
  This dataset was used to finetune this model.
28
 
29
+ ### Project Description
30
+ This project aims to use off-the-shelf large language models for text-to-SQL program sysnthesis tasks. After experimenting with various models, fine-tuning hyperparameters, and training datasets an optimal solution was identified by fine-tuning the [WizardLM/WizardCoder-15B-V1.0](https://huggingface.co/WizardLM/WizardCoder-15B-V1.0) base model using QLoRA techniques on [this](https://huggingface.co/datasets/richardr1126/spider-context-validation) customized Spider training dataset. The resultant model, [richardr1126/spider-skeleton-wizard-coder-merged](richardr1126/spider-skeleton-wizard-coder-merged), demonstrates **61% execution accuracy** when evaluated. The project utilizes a custom validation dataset that incorporates database context into the question. A live demonstration of the model is available on Hugging Face Space, facilitated by the Gradio library for user-friendly GUI.
31
+
32
+ Note: You might have to wake the Space up if it is sleeping, should take less than 10 minutes.
33
+ ### Spider Skeleton WizardCoder - [test-suite-sql-eval](https://github.com/taoyds/test-suite-sql-eval) results
34
+ With temperature set to 0.0, top_p set to 0.9, and top_k set to 0, the model achieves **61% execution accuracy** on the Spider test suite.
35
+
36
+ <img src="https://raw.githubusercontent.com/cuplv/text-to-sql-wizardcoder/main/eval/plots/spiderwizard-plus-chatgpt.svg" height="300">
37
+ <img src="https://raw.githubusercontent.com/cuplv/text-to-sql-wizardcoder/main/eval/plots/spiderwizard-vs-chatgpt.svg" height="300">
38
+
39
+ Note: ChatGPT was evaluated with the default hyperparameters and with the system message `You are a sophisticated AI assistant capable of converting text into SQL queries. You can only output SQL, don't add any other text.`
40
+
41
  ## Citation
42
 
43
  Please cite the repo if you use the data or code in this repo.