yoniebans/starcoder2-3b-qlora-solidity

Model Card for Starcoder2-3B Fine-Tuned with QLORA on Solidity Dataset

Model Details

Model Description

This model has been fine-tuned using QLORA specifically for generating Solidity smart contracts from natural language instructions.

• Developed by: @yoniebans
• Model type: Transformer-based Causal Language Model
• Language(s) (NLP): English with programming language syntax (Solidity)
• License: [More Information Needed]
• Finetuned from model bigcode/starcoder2-3b

Model Sources [optional]

• Repository: [More Information Needed]
• Demo [optional]: [More Information Needed]

Uses

Direct Use

This model is designed to demonstrate how fine-tuning a base model such as starcoder2-3b using qlora with a targetted dataset can increase the model's ability to create Solidity code directly from natural language descriptions or instructions.

Out-of-Scope Use

This model is not intended for:

• Deployment in production systems without rigorous testing.
• Use in non-technical text generation or any context outside smart contract development.

Bias, Risks, and Limitations

The training data primarily consists of code from publicly sourced Solidity projects which may not encompass the full diversity of programming styles and techniques.

Recommendations

Users are advised to use this model as a starting point for development and not as a definitive solution. Generated code should always be reviewed by experienced developers to ensure security and functionality.

How to Get Started with the Model

Use the code below to get started with the model.

import sys
from peft import PeftModel
from transformers import BitsAndBytesConfig, AutoTokenizer, AutoModelForCausalLM
import torch
import accelerate

use_4bit = True
bnb_4bit_compute_dtype = "float32"
bnb_4bit_quant_type = "nf4"
use_double_nested_quant = True
compute_dtype = getattr(torch, bnb_4bit_compute_dtype)

device_map = "auto"
max_memory = '45000MB'
n_gpus = torch.cuda.device_count()
max_memory = {i: max_memory for i in range(n_gpus)}

model = AutoModelForCausalLM.from_pretrained(
    "bigcode/starcoder2-3b",
    cache_dir=None,
    device_map=device_map,
    max_memory=max_memory,
    quantization_config=BitsAndBytesConfig(
        load_in_4bit=use_4bit,
        llm_int8_threshold=6.0,
        llm_int8_has_fp16_weight=False,
        bnb_4bit_compute_dtype=compute_dtype,
        bnb_4bit_use_double_quant=use_double_nested_quant,
        bnb_4bit_quant_type=bnb_4bit_quant_type
    ),
    torch_dtype=torch.float32,
    trust_remote_code=False    
)
adapter_weights = "yoniebans/starcoder2-3b-qlora-solidity"
model = PeftModel.from_pretrained(model, adapter_weights)

tokenizer = AutoTokenizer.from_pretrained(
        checkpoint,
        cache_dir=None,
        padding_side="right",
        use_fast=False,
        tokenizer_type=None, # Needed for HF name change
        trust_remote_code=False,
        use_auth_token=False,
    )


input='Make a smart contract for a memecoin named 'LLMAI', adhering to the ERC20 standard. The contract should enforce a purchase limit where no individual wallet can acquire more than 1% of the total token supply, which is set at 10 billion tokens. This purchasing limit should be modifiable and can only be disabled by the contract owner at their discretion. Note that the interfaces for ERC20, Ownable, and any other dependencies should be assumed as already imported and do not need to be included in your code response.'

prompt = f"""### Instruction:
Use the Task below and the Input given to write the Response, which is a programming code that can solve the following Task:

### Task:
{input}

### Solution:
"""

input_ids = tokenizer(prompt, return_tensors="pt", truncation=True).input_ids.cuda()
outputs = model.generate(
    input_ids=input_ids, 
    max_new_tokens=2048, 
    do_sample=True, 
    top_p=0.9, 
    temperature=0.001, 
    pad_token_id=1
)

output_text = tokenizer.batch_decode(outputs.detach().cpu().numpy(), skip_special_tokens=True)[0]
output_text_without_prompt = output_text[len(prompt):]

file_path = './smart_contract.sol'

with open(file_path, 'w') as file:
    file.write(output_text_without_prompt)

print(f"Output written to {file_path}")

Training Details

Training Data

The model was trained on a dataset consisting of pairs of natural language instructions and their corresponding Solidity code implementations. This dataset includes over 6000 input and outputs.

Training Procedure

Training Hyperparameters

• Number of train epochs: 30
• Double quant: true
• Quant type: nf4
• Bits: 4
• Lora r: 64
• Lora aplha: 16
• Lora dropout: 0.0
• Per device train batch size: 1
• Gradient accumulation steps: 1
• Max steps: 10000
• Weight decay: 0.0 #use lora dropout instead for regularization if needed
• Learning rate: 2e-4
• Max gradient normal: 0.3
• Gradient checkpointing: true
• FP16: false
• FP16 option level: O1
• BF16: false
• Optimizer: paged AdamW 32-bit
• Learning rate scheduler type: constant
• Warmup ratio: 0.03

Speeds, Sizes, Times [optional]

Epoch	Grad Norm	Loss	Step
0.03	0.513248	2.0948	10
0.59	0.976398	0.7952	200
1.18	0.291436	0.5081	400
1.78	0.197318	0.5007	600
2.37	0.143200	0.4189	800
2.96	0.153539	0.4314	1000
3.55	0.199708	0.4554	1200
4.15	0.258209	0.4856	1400
4.74	0.295185	0.5446	1600
5.33	0.157903	0.3209	1800
5.92	0.149965	0.3203	2000
6.52	0.166200	0.4046	2200
7.11	0.181177	0.2157	2400
7.70	0.156556	0.2937	2600
8.29	0.309143	0.2928	2800
8.89	0.211304	0.3414	3000
9.48	0.166119	0.2716	3200
10.07	0.265576	0.1727	3400
10.66	0.231398	0.2219	3600
11.26	0.183477	0.1706	3800
11.85	0.182451	0.1471	4000
12.44	0.272920	0.1793	4200
13.03	0.189667	0.1241	4400
13.62	0.136364	0.1278	4600
14.22	0.297066	0.1043	4800
14.81	0.213272	0.1760	5000
15.40	0.162718	0.1062	5200
15.99	0.229559	0.1012	5400
16.59	0.271037	0.1180	5600
17.18	0.239012	0.0871	5800
17.77	0.175727	0.0894	6000
18.36	0.151963	0.1154	6200
18.96	0.202392	0.1096	6400
19.55	0.202703	0.0764	6600
20.18	0.148534	0.0551	6800
20.77	0.151745	0.0599	7000
21.36	0.188332	0.0707	7200
21.95	0.190111	0.0987	7400
22.55	0.163640	0.0567	7600
23.14	0.209854	0.0534	7800
23.73	0.225155	0.0626	8000
24.32	0.172093	0.0389	8200
24.92	0.162895	0.0347	8400
25.52	0.187249	0.0592	8600
26.11	0.182813	0.0368	8800
26.70	0.243531	0.0427	9000
27.29	0.190624	0.0325	9200
27.89	0.189085	0.0311	9400
28.48	0.171129	0.0322	9600
29.07	0.138704	0.0353	9800
29.66	0.216177	0.0312	10000

Results

Summary

Initial evaluations show promising results in generating Solidity code, with further testing planned.

Environmental Impact

• Hardware Type: NVIDIA A40 GPU, 48GB VRAM, 48GB RAM, 9vCPU
• Hours used: 18
• Cloud Provider: runpod.io
• Compute Region: [More Information Needed]
• Carbon Emitted: [More Information Needed]

Citation [optional]

BibTeX:

@misc{lozhkov2024starcoder,
      title={StarCoder 2 and The Stack v2: The Next Generation}, 
      author={Anton Lozhkov and Raymond Li and Loubna Ben Allal and Federico Cassano and Joel Lamy-Poirier and Nouamane Tazi and Ao Tang and Dmytro Pykhtar and Jiawei Liu and Yuxiang Wei and Tianyang Liu and Max Tian and Denis Kocetkov and Arthur Zucker and Younes Belkada and Zijian Wang and Qian Liu and Dmitry Abulkhanov and Indraneil Paul and Zhuang Li and Wen-Ding Li and Megan Risdal and Jia Li and Jian Zhu and Terry Yue Zhuo and Evgenii Zheltonozhskii and Nii Osae Osae Dade and Wenhao Yu and Lucas Krauß and Naman Jain and Yixuan Su and Xuanli He and Manan Dey and Edoardo Abati and Yekun Chai and Niklas Muennighoff and Xiangru Tang and Muhtasham Oblokulov and Christopher Akiki and Marc Marone and Chenghao Mou and Mayank Mishra and Alex Gu and Binyuan Hui and Tri Dao and Armel Zebaze and Olivier Dehaene and Nicolas Patry and Canwen Xu and Julian McAuley and Han Hu and Torsten Scholak and Sebastien Paquet and Jennifer Robinson and Carolyn Jane Anderson and Nicolas Chapados and Mostofa Patwary and Nima Tajbakhsh and Yacine Jernite and Carlos Muñoz Ferrandis and Lingming Zhang and Sean Hughes and Thomas Wolf and Arjun Guha and Leandro von Werra and Harm de Vries},
      year={2024},
      eprint={2402.19173},
      archivePrefix={arXiv},
      primaryClass={cs.SE}
}

APA:

Starcoder2-3B: BigCode Team. (2024). Starcoder2-3B [Software]. Available from https://huggingface.co/bigcode/starcoder2-3b

AlfredPros Smart Contracts Instructions: AlfredPros. (2023). Smart Contracts Instructions [Data set]. Available from https://huggingface.co/datasets/AlfredPros/smart-contracts-instructions

Model Card Contact

jonny@edennetwork.io