ruslanmv commited on
Commit
1351e37
1 Parent(s): 8343ab6

Create README.md

Browse files
Files changed (1) hide show
  1. README.md +137 -0
README.md ADDED
@@ -0,0 +1,137 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ language:
3
+ - en
4
+ - it
5
+ license: apache-2.0
6
+ tags:
7
+ - text-generation-inference
8
+ - transformers
9
+ - ruslanmv
10
+ - llama
11
+ - trl
12
+ - sft
13
+ ---
14
+ # Meta-Llama 3.1 8B Text-to-SQL GPTQ Model
15
+
16
+ This repository provides a quantized 8-billion-parameter Meta-Llama model fine-tuned for text-to-SQL tasks. The model is optimized with GPTQ quantization for efficient inference. Below you'll find instructions to load, use, and fine-tune the model.
17
+
18
+ ## Model Details
19
+
20
+ - **Model Size**: 8B
21
+ - **Quantization**: GPTQ (4-bit)
22
+ - **Languages Supported**: English, Italian
23
+ - **Task**: Text-to-SQL generation
24
+ - **License**: Apache 2.0
25
+
26
+ ## Installation Requirements
27
+
28
+ Before using the model, ensure that you have the following dependencies installed. We recommend using the same versions to avoid any compatibility issues.
29
+
30
+ ```bash
31
+ # Install the required PyTorch version with CUDA support (ensure CUDA 12.1 is installed)
32
+ !pip install torch==2.2.1 torchvision==0.17.1 torchaudio==2.2.1 --index-url https://download.pytorch.org/whl/cu121
33
+
34
+ # Install AutoGPTQ for quantized model handling
35
+ !pip install auto-gptq --no-build-isolation
36
+
37
+ # Install Optimum for model optimization
38
+ !pip install optimum
39
+ ```
40
+
41
+ After installing the dependencies, reset your instance to ensure everything works correctly.
42
+
43
+ ## Loading the Model
44
+
45
+ To load the quantized Meta-Llama 3.1 model and use it for text-to-SQL tasks, use the following Python code:
46
+
47
+ ```python
48
+ from transformers import AutoTokenizer, pipeline
49
+ from auto_gptq import AutoGPTQForCausalLM
50
+ import torch
51
+
52
+ # Define the Alpaca-style prompt template
53
+ alpaca_prompt = """Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request.
54
+
55
+ ### Instruction:
56
+ {}
57
+
58
+ ### Input:
59
+ {}
60
+
61
+ ### Response:
62
+ """
63
+
64
+ # Model directory and tokenizer
65
+ quantized_model_dir = "meta-llama-8b-quantized-4bit" # Path where quantized model is saved
66
+ tokenizer = AutoTokenizer.from_pretrained(quantized_model_dir)
67
+
68
+ # Load the quantized model
69
+ model = AutoGPTQForCausalLM.from_quantized(
70
+ quantized_model_dir,
71
+ device_map="auto", # Automatically map the model to the available device (GPU or CPU)
72
+ torch_dtype=torch.float16, # Ensure FP16 for efficiency
73
+ use_safetensors=True # If you saved the model using safetensors format, set this to True
74
+ )
75
+
76
+ # Set up the text generation pipeline without specifying the device
77
+ pipeline = pipeline(
78
+ "text-generation",
79
+ model=model,
80
+ tokenizer=tokenizer
81
+ )
82
+
83
+ # Function to generate SQL query from input text using the Alpaca prompt
84
+ def generate_sql(input_text):
85
+ # Format the prompt
86
+ prompt = alpaca_prompt.format(
87
+ "Provide the SQL query",
88
+ input_text
89
+ )
90
+
91
+ # Generate the response using the pipeline
92
+ generated_text = pipeline(
93
+ prompt,
94
+ max_length=200,
95
+ eos_token_id=tokenizer.eos_token_id
96
+ )[0]["generated_text"]
97
+
98
+ # Clean the output by removing the prompt and any extra newlines
99
+ cleaned_output = generated_text.replace(prompt, '').strip()
100
+
101
+ return cleaned_output
102
+
103
+ # Example usage
104
+ italian_input = "Seleziona tutte le colonne della tabella table1 dove la colonna anni è uguale a 2020"
105
+ sql_query = generate_sql(italian_input)
106
+ print(sql_query)
107
+ ```
108
+
109
+ ## Example Usage
110
+
111
+ The example script shows how to generate SQL queries from natural language text. Simply provide a request in Italian or English, and the model will generate an appropriate SQL query.
112
+
113
+ Example input:
114
+
115
+ ```python
116
+ italian_input = "Seleziona tutte le colonne della tabella table1 dove la colonna anni è uguale a 2020"
117
+ sql_query = generate_sql(italian_input)
118
+ print(sql_query)
119
+ ```
120
+
121
+ Example output:
122
+
123
+ ```sql
124
+ SELECT * FROM table1 WHERE anni = 2020;
125
+ ```
126
+
127
+ ## Model Tags
128
+
129
+ - **text-generation-inference**
130
+ - **transformers**
131
+ - **llama**
132
+ - **trl**
133
+ - **sft**
134
+
135
+ ## License
136
+
137
+ This model is released under the [Apache License 2.0](LICENSE).