Init

.gitattributes

@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+pytorch_model.bin filter=lfs diff=lfs merge=lfs -text

README.md

@@ -1,3 +1,96 @@
 ---
+language:
+- en
+library_name: transformers
+pipeline_tag: text-generation
+datasets:
+- ise-uiuc/Magicoder-OSS-Instruct-75K
+- ise-uiuc/Magicoder-Evol-Instruct-110K
+tags:
+- code
 license: apache-2.0
+model-index:
+- name: SpeechlessCoder
+  results:
+  - task:
+      type: text-generation
+    dataset:
+      type: openai_humaneval
+      name: HumanEval
+    metrics:
+    - name: pass@1
+      type: pass@1
+      value: 
+      verified: false
 ---
+
+<p><h1> speechless-coder-ds-1.3b  </h1></p>
+
+Use the following dataset to fine-tune deepseek-ai/deepseek-coder-1.3b in order to improve the model's reasoning and planning abilities.
+
+context window length: 8192
+max_tokens > 128 && < 8192
+>
+Total 185,193 samples 426 MB
+- ise-uiuc/Magicoder-OSS-Instruct-75K 75,186 samples
+- ise-uiuc/Magicoder-Evol-Instruct-110K 110,007 samples
+
+
+50 samples/T=0.2/MaxTokens=512/Top_P=0.95
+
+Code: https://github.com/uukuguy/speechless
+
+## HumanEval
+
+| Metric | Value |
+| --- | --- |
+| humaneval-python |  |
+
+[Big Code Models Leaderboard](https://huggingface.co/spaces/bigcode/bigcode-models-leaderboard)
+
+CodeLlama-34B-Python: 53.29
+
+CodeLlama-34B-Instruct: 50.79
+
+CodeLlama-13B-Instruct: 50.6
+
+CodeLlama-34B: 45.11
+
+CodeLlama-13B-Python: 42.89
+
+CodeLlama-13B: 35.07
+
+## BigCode Eval
+0.205055
+
+- metrics_humanevalfixtests-cpp:    "pass@1": 0.054878048780487805
+- metrics_humanevalfixtests-go:    "pass@1": 0.054878048780487805
+- metrics_humanevalfixtests-java:    "pass@1": 0.042682926829268296
+- metrics_humanevalfixtests-js:    "pass@1": 0.0975609756097561
+- metrics_humanevalfixtests-python:    "pass@1": 0.06707317073170732
+- metrics_humanevalfixtests-rust:    "pass@1": 0.018292682926829267
+
+0.332906
+
+- metrics_humanevalsynthesize-cpp:    "pass@1": 0.3475609756097561
+- metrics_humanevalsynthesize-go:    "pass@1": 0.25609756097560976
+- metrics_humanevalsynthesize-java:    "pass@1": 0.3353658536585366
+- metrics_humanevalsynthesize-js:    "pass@1": 0.35365853658536583
+- metrics_humanevalsynthesize-python:    "pass@1": 0.4024390243902439
+- metrics_humanevalsynthesize-rust:    "pass@1": 0.20121951219512196
+- metrics_mbpp:    "pass@1": 0.434
+
+
+
+
+## LMEval
+
+[Open LLM Leaderboard](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard)
+| Metric | Value |
+| --- | --- |
+| ARC | |
+| HellaSwag | |
+| MMLU | |
+| TruthfulQA |  |
+| Average |  |
+