doberst commited on
Commit
9075c24
1 Parent(s): b95e4e4

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +25 -2
README.md CHANGED
@@ -6,12 +6,32 @@ license: apache-2.0
6
 
7
  <!-- Provide a quick summary of what the model is/does. -->
8
 
9
- BLING-1.4b-0.1 is the first model release in the BLING ("Best Little Instruction-following No-GPU-required") model series.
10
 
11
  BLING models are fine-tuned with distilled high-quality custom instruct datasets, targeted at a specific subset of instruct tasks with
12
  the objective of providing a high-quality Instruct model that is 'inference-ready' on a CPU laptop even
13
  without using any advanced quantization optimizations.
14
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
15
  ### Model Description
16
 
17
  <!-- Provide a longer summary of what this model is. -->
@@ -55,6 +75,9 @@ without the need for a lot of complex instruction verbiage - provide a text pass
55
 
56
  Any model can provide inaccurate or incomplete information, and should be used in conjunction with appropriate safeguards and fact-checking mechanisms.
57
 
 
 
 
58
 
59
  ## How to Get Started with the Model
60
 
@@ -97,7 +120,7 @@ BLING models are built on top of EleutherAI/Pythia base - please see citation fo
97
 
98
  Darren Oberst & llmware team
99
 
100
- Please reach out anytime if you are interested in this project and would like to participate and work with us!
101
 
102
 
103
 
 
6
 
7
  <!-- Provide a quick summary of what the model is/does. -->
8
 
9
+ BLING-1.4b-0.1 is part of the BLING ("Best Little Instruction-following No-GPU-required") model series.
10
 
11
  BLING models are fine-tuned with distilled high-quality custom instruct datasets, targeted at a specific subset of instruct tasks with
12
  the objective of providing a high-quality Instruct model that is 'inference-ready' on a CPU laptop even
13
  without using any advanced quantization optimizations.
14
 
15
+
16
+ ### Benchmark Tests
17
+
18
+ Evaluated against the benchmark test: [RAG-Instruct-Benchmark-Tester](https://www.huggingface.co/datasets/llmware/rag_instruct_benchmark_tester)
19
+ Average of 2 Test Runs with 1 point for correct answer, 0.5 point for partial correct or blank / NF, 0.0 points for incorrect, and -1 points for hallucinations.
20
+
21
+ --**Accuracy Score**: **82.25** correct out of 100
22
+ --Not Found Classification: 40.0%
23
+ --Boolean: 61.25%
24
+ --Math/Logic: 8.75%
25
+ --Complex Questions (1-5): 1 (Low)
26
+ --Summarization Quality (1-5): 2 (Coherent, extractive)
27
+ --Hallucinations: No hallucinations observed in test runs.
28
+
29
+ For test run results (and good indicator of target use cases), please see the files ("core_rag_test" and "answer_sheet" in this repo).
30
+
31
+ --As a reference point, this model shows substantial improvements in results, compared with the BLING 1.0B Pythia, with fine-tuning and the base training substantially the same.
32
+ --The model's ability to follow instructions and answer detailed questions improves dramatically from 1.0B -> 1.4B parameters.
33
+
34
+
35
  ### Model Description
36
 
37
  <!-- Provide a longer summary of what this model is. -->
 
75
 
76
  Any model can provide inaccurate or incomplete information, and should be used in conjunction with appropriate safeguards and fact-checking mechanisms.
77
 
78
+ Please refer to the benchmark score and testing results for indicator as to the applicability of this model to your intended use case.
79
+ We have found that this model is reasonably effective and accurate for fact-based, extractive tasks, including key-value, question-answering, and basic summarization.
80
+
81
 
82
  ## How to Get Started with the Model
83
 
 
120
 
121
  Darren Oberst & llmware team
122
 
123
+ Please reach out anytime if you are interested in this project!
124
 
125
 
126