JRosenkranz commited on
Commit
2e7a11a
1 Parent(s): 0c0c1ce

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +20 -0
README.md CHANGED
@@ -34,6 +34,26 @@ The model was trained on 2T tokens, with sampling proportions designed to match
34
  | PG19 | A repository of free e-books with focus on older works for which U.S. copyright has expired. | 0.25% | https://github.com/google-deepmind/pg19 |
35
  | Webhose | Unstructured web content converted into machine-readable data feeds purchased by IBM. | 5% | N/A |
36
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
37
  **Bias, Risks, and Limitations**
38
 
39
  Granite-7b-base is a base model and has not undergone any safety alignment, there it may produce problematic outputs. In the absence of adequate safeguards and RLHF, there exists a risk of malicious utilization of these models for generating disinformation or harmful content. Caution is urged against complete reliance on a specific language model for crucial decisions or impactful information, as preventing these models from fabricating content is not straightforward. Additionally, it remains uncertain whether smaller models might exhibit increased susceptibility to hallucination in ungrounded generation scenarios due to their reduced sizes and memorization capacities. This aspect is currently an active area of research, and we anticipate more rigorous exploration, comprehension, and mitigations in this domain.
 
34
  | PG19 | A repository of free e-books with focus on older works for which U.S. copyright has expired. | 0.25% | https://github.com/google-deepmind/pg19 |
35
  | Webhose | Unstructured web content converted into machine-readable data feeds purchased by IBM. | 5% | N/A |
36
 
37
+ **Evaluation Results**
38
+
39
+ LM-eval Harness Scores
40
+
41
+ | Evaluation metric | Llama2-7B (baseline) | Granite-7b-base |
42
+ |----------------------------|----------------------|-----------------|
43
+ | MMLU (zero shot) | 0.41 | 0.43 |
44
+ | MMLU (5-shot weighted avg) | 0.47 | 0.50 |
45
+ | Arc challenge | 0.46 | 0.44 |
46
+ | Arc easy | 0.74 | 0.71 |
47
+ | Boolq | 0.78 | 0.76 |
48
+ | Copa | 0.87 | 0.83 |
49
+ | Hellaswag | 0.76 | 0.74 |
50
+ | Openbookqa | 0.44 | 0.42 |
51
+ | Piqa | 0.79 | 0.79 |
52
+ | Sciq | 0.91 | 0.91 |
53
+ | Winogrande | 0.69 | 0.67 |
54
+ | Truthfulqa | 0.39 | 0.39 |
55
+ | GSM8k (8-shot) | 0.13 | 0.11 |
56
+
57
  **Bias, Risks, and Limitations**
58
 
59
  Granite-7b-base is a base model and has not undergone any safety alignment, there it may produce problematic outputs. In the absence of adequate safeguards and RLHF, there exists a risk of malicious utilization of these models for generating disinformation or harmful content. Caution is urged against complete reliance on a specific language model for crucial decisions or impactful information, as preventing these models from fabricating content is not straightforward. Additionally, it remains uncertain whether smaller models might exhibit increased susceptibility to hallucination in ungrounded generation scenarios due to their reduced sizes and memorization capacities. This aspect is currently an active area of research, and we anticipate more rigorous exploration, comprehension, and mitigations in this domain.