Create README.md

README.md

@@ -0,0 +1,121 @@
+---
+inference: false
+datasets:
+- bigcode/commitpackft
+model-index:
+  - name: patched-coder-34b
+    results:
+      - task:
+          type: text-generation
+        dataset:
+          type: openai_humaneval
+          name: HumanEval
+        metrics:
+          - name: pass@1
+            type: pass@1
+            value: 53.567
+            verified: false
+      - task:
+          type: text-generation
+        dataset:
+          type: bigcode/humanevalpack
+          name: HumanEvalFix Python
+        metrics:
+          - name: pass@1
+            type: pass@1
+            value: 41.341
+            verified: false
+      - task:
+          type: text-generation
+        dataset:
+          type: patched-codes/static-analysis-eval
+          name: Static Analysis Eval
+        metrics:
+          - name: pass@1
+            type: pass@1
+            value: 51.316
+            verified: false
+---
+# Model Card for patched-coder-34b
+
+
+This is an instruction fine-tuned model focussed on the task of patching code. Patching may include fixing bugs, remediating security vulnerabilities, 
+doing API migrations and other kinds of code matainence.
+
+## Model Details
+
+### Model Description
+
+- **Developed by:** [codelion](https://huggingface.co/codelion)
+- **Model type:** Code Llama
+- **Finetuned from model:** [CodeLlama-34b-Python](https://huggingface.co/codellama/CodeLlama-34b-Python-hf)
+
+
+## How to Get Started with the Model
+
+Make sure to install Transformers from the main git branch:
+
+```bash
+pip install git+https://github.com/huggingface/transformers.git
+```
+
+## How to Prompt the Model
+
+This model accepts the alpaca instruction format.
+
+For example: 
+
+```
+### Instruction:
+{instruction}
+
+### Input:
+{input}
+
+### Response:
+...
+```
+
+## Bias, Risks, and Limitations
+
+This model has undergone very limited testing. Additional safety testing should be performed before any real-world deployments.
+
+## Training Details
+
+- **GPU:** A100 80 GB
+- **Time:** ~8 hrs
+
+### Training Data
+
+The model was fine-tuned on [commitpackft](https://huggingface.co/datasets/bigcode/commitpackft), an open dataset consisting of commits.
+We started with the commits for the `python` langauge from the dataset and then filtered all the commits that were related to fixing bugs. 
+
+### Training Procedure 
+
+Instruction fine-tuning to follow instructions in natural langauge related to code. We load the quantized base model in 4 bits 
+and then use QLoRA for Parameter-Efficient Fine-Tuning (PEFT) with Flash Attention. The model was trained for 2 epochs.
+
+#### Training Hyperparameters
+
+**Training regime:** 
+
+The following `bitsandbytes` quantization config was used during training:
+  - quant_method: bitsandbytes
+  - load_in_8bit: False
+  - load_in_4bit: True
+  - llm_int8_threshold: 6.0
+  - llm_int8_skip_modules: None
+  - llm_int8_enable_fp32_cpu_offload: False
+  - llm_int8_has_fp16_weight: False
+  - bnb_4bit_quant_type: nf4
+  - bnb_4bit_use_double_quant: True
+  - bnb_4bit_compute_dtype: bfloat16
+
+## Evaluation
+
+We evaluate the model on `HumanEval` and `HumanEvalPack` benchmarks using 
+[Code Generation LM Evaluation Harness](https://github.com/bigcode-project/bigcode-evaluation-harness).
+
+We also evaluate the model for vulnerability remediation using the `Static Analysis Eval` benchmark available [here](https://huggingface.co/datasets/patched-codes/static-analysis-eval).
+
+### Results