Create README.md
Browse files
README.md
ADDED
@@ -0,0 +1,137 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
1 |
+
---
|
2 |
+
language:
|
3 |
+
- en
|
4 |
+
- it
|
5 |
+
license: apache-2.0
|
6 |
+
tags:
|
7 |
+
- text-generation-inference
|
8 |
+
- transformers
|
9 |
+
- ruslanmv
|
10 |
+
- llama
|
11 |
+
- trl
|
12 |
+
- sft
|
13 |
+
---
|
14 |
+
# Meta-Llama 3.1 8B Text-to-SQL GPTQ Model
|
15 |
+
|
16 |
+
This repository provides a quantized 8-billion-parameter Meta-Llama model fine-tuned for text-to-SQL tasks. The model is optimized with GPTQ quantization for efficient inference. Below you'll find instructions to load, use, and fine-tune the model.
|
17 |
+
|
18 |
+
## Model Details
|
19 |
+
|
20 |
+
- **Model Size**: 8B
|
21 |
+
- **Quantization**: GPTQ (4-bit)
|
22 |
+
- **Languages Supported**: English, Italian
|
23 |
+
- **Task**: Text-to-SQL generation
|
24 |
+
- **License**: Apache 2.0
|
25 |
+
|
26 |
+
## Installation Requirements
|
27 |
+
|
28 |
+
Before using the model, ensure that you have the following dependencies installed. We recommend using the same versions to avoid any compatibility issues.
|
29 |
+
|
30 |
+
```bash
|
31 |
+
# Install the required PyTorch version with CUDA support (ensure CUDA 12.1 is installed)
|
32 |
+
!pip install torch==2.2.1 torchvision==0.17.1 torchaudio==2.2.1 --index-url https://download.pytorch.org/whl/cu121
|
33 |
+
|
34 |
+
# Install AutoGPTQ for quantized model handling
|
35 |
+
!pip install auto-gptq --no-build-isolation
|
36 |
+
|
37 |
+
# Install Optimum for model optimization
|
38 |
+
!pip install optimum
|
39 |
+
```
|
40 |
+
|
41 |
+
After installing the dependencies, reset your instance to ensure everything works correctly.
|
42 |
+
|
43 |
+
## Loading the Model
|
44 |
+
|
45 |
+
To load the quantized Meta-Llama 3.1 model and use it for text-to-SQL tasks, use the following Python code:
|
46 |
+
|
47 |
+
```python
|
48 |
+
from transformers import AutoTokenizer, pipeline
|
49 |
+
from auto_gptq import AutoGPTQForCausalLM
|
50 |
+
import torch
|
51 |
+
|
52 |
+
# Define the Alpaca-style prompt template
|
53 |
+
alpaca_prompt = """Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request.
|
54 |
+
|
55 |
+
### Instruction:
|
56 |
+
{}
|
57 |
+
|
58 |
+
### Input:
|
59 |
+
{}
|
60 |
+
|
61 |
+
### Response:
|
62 |
+
"""
|
63 |
+
|
64 |
+
# Model directory and tokenizer
|
65 |
+
quantized_model_dir = "meta-llama-8b-quantized-4bit" # Path where quantized model is saved
|
66 |
+
tokenizer = AutoTokenizer.from_pretrained(quantized_model_dir)
|
67 |
+
|
68 |
+
# Load the quantized model
|
69 |
+
model = AutoGPTQForCausalLM.from_quantized(
|
70 |
+
quantized_model_dir,
|
71 |
+
device_map="auto", # Automatically map the model to the available device (GPU or CPU)
|
72 |
+
torch_dtype=torch.float16, # Ensure FP16 for efficiency
|
73 |
+
use_safetensors=True # If you saved the model using safetensors format, set this to True
|
74 |
+
)
|
75 |
+
|
76 |
+
# Set up the text generation pipeline without specifying the device
|
77 |
+
pipeline = pipeline(
|
78 |
+
"text-generation",
|
79 |
+
model=model,
|
80 |
+
tokenizer=tokenizer
|
81 |
+
)
|
82 |
+
|
83 |
+
# Function to generate SQL query from input text using the Alpaca prompt
|
84 |
+
def generate_sql(input_text):
|
85 |
+
# Format the prompt
|
86 |
+
prompt = alpaca_prompt.format(
|
87 |
+
"Provide the SQL query",
|
88 |
+
input_text
|
89 |
+
)
|
90 |
+
|
91 |
+
# Generate the response using the pipeline
|
92 |
+
generated_text = pipeline(
|
93 |
+
prompt,
|
94 |
+
max_length=200,
|
95 |
+
eos_token_id=tokenizer.eos_token_id
|
96 |
+
)[0]["generated_text"]
|
97 |
+
|
98 |
+
# Clean the output by removing the prompt and any extra newlines
|
99 |
+
cleaned_output = generated_text.replace(prompt, '').strip()
|
100 |
+
|
101 |
+
return cleaned_output
|
102 |
+
|
103 |
+
# Example usage
|
104 |
+
italian_input = "Seleziona tutte le colonne della tabella table1 dove la colonna anni è uguale a 2020"
|
105 |
+
sql_query = generate_sql(italian_input)
|
106 |
+
print(sql_query)
|
107 |
+
```
|
108 |
+
|
109 |
+
## Example Usage
|
110 |
+
|
111 |
+
The example script shows how to generate SQL queries from natural language text. Simply provide a request in Italian or English, and the model will generate an appropriate SQL query.
|
112 |
+
|
113 |
+
Example input:
|
114 |
+
|
115 |
+
```python
|
116 |
+
italian_input = "Seleziona tutte le colonne della tabella table1 dove la colonna anni è uguale a 2020"
|
117 |
+
sql_query = generate_sql(italian_input)
|
118 |
+
print(sql_query)
|
119 |
+
```
|
120 |
+
|
121 |
+
Example output:
|
122 |
+
|
123 |
+
```sql
|
124 |
+
SELECT * FROM table1 WHERE anni = 2020;
|
125 |
+
```
|
126 |
+
|
127 |
+
## Model Tags
|
128 |
+
|
129 |
+
- **text-generation-inference**
|
130 |
+
- **transformers**
|
131 |
+
- **llama**
|
132 |
+
- **trl**
|
133 |
+
- **sft**
|
134 |
+
|
135 |
+
## License
|
136 |
+
|
137 |
+
This model is released under the [Apache License 2.0](LICENSE).
|