ericzzz commited on
Commit
08b5e87
β€’
1 Parent(s): bb5f861

Update README.md

Browse files

Update evaluation results

Files changed (1) hide show
  1. README.md +24 -9
README.md CHANGED
@@ -14,13 +14,28 @@ tags:
14
 
15
  Falcon-RW-1B-Instruct-OpenOrca is a 1B parameter, causal decoder-only model based on [Falcon-RW-1B](https://huggingface.co/tiiuae/falcon-rw-1b) and finetuned on the [Open-Orca/SlimOrca](https://huggingface.co/datasets/Open-Orca/SlimOrca) dataset.
16
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
17
  **πŸš€ Motivations**
18
- 1. To create a smaller, open-source, instruction-finetuned, ready-to-use model accessible for users with limited computational resources; to make a decent ready-to-use model for running on lower-end consumer GPUs.
19
  2. To harness the strength of Falcon-RW-1B, a competitive model in its own right, and enhance its capabilities with instruction finetuning.
20
 
21
  ## πŸ“– How to Use
22
 
23
- The model operates with a structured prompt format, incorporating `<SYS>`, `<INST>`, and `<RESP>` tags to demarcate different parts of the input. The system message and instruction are placed within these tags, with the `<RESP>` tag triggering the model's response.
24
 
25
  ### πŸ“ Example Code
26
 
@@ -33,16 +48,16 @@ model = 'ericzzz/falcon-rw-1b-instruct-openorca'
33
 
34
  tokenizer = AutoTokenizer.from_pretrained(model)
35
  pipeline = transformers.pipeline(
36
- "text-generation",
37
  model=model,
38
  tokenizer=tokenizer,
39
  torch_dtype=torch.bfloat16,
40
- device_map="auto",
41
  )
42
 
43
- system_message = "You are a helpful assistant. Give short answers."
44
- instruction = "What is AI? Give some examples."
45
- prompt = f"<SYS> {system_message} <INST> {instruction} <RESP> "
46
 
47
  response = pipeline(
48
  prompt,
@@ -54,5 +69,5 @@ print(response[0]['generated_text'])
54
  # AI, or Artificial Intelligence, refers to the ability of machines and software to perform tasks that require human intelligence, such as learning, reasoning, and problem-solving. It can be used in various fields like computer science, engineering, medicine, and more. Some common applications include image recognition, speech translation, and natural language processing.
55
  ```
56
 
57
- ## Contact
58
- eric.fu96@aol.com
 
14
 
15
  Falcon-RW-1B-Instruct-OpenOrca is a 1B parameter, causal decoder-only model based on [Falcon-RW-1B](https://huggingface.co/tiiuae/falcon-rw-1b) and finetuned on the [Open-Orca/SlimOrca](https://huggingface.co/datasets/Open-Orca/SlimOrca) dataset.
16
 
17
+ **πŸ“Š Evaluation Results**
18
+
19
+ Falcon-RW-1B-Instruct-OpenOrca is the #1 ranking model on [Open LLM Leaderboard](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard) in ~1.5B parameters category!
20
+
21
+ | Metric | falcon-rw-1b-instruct-openorca | falcon-rw-1b |
22
+ |------------|-------------------------------:|-------------:|
23
+ | ARC | 34.56 | 35.07 |
24
+ | HellaSwag | 60.93 | 63.56 |
25
+ | MMLU | 28.77 | 25.28 |
26
+ | TruthfulQA | 37.42 | 35.96 |
27
+ | Winogrande | 60.69 | 62.04 |
28
+ | GSM8K | 1.21 | 0.53 |
29
+ | DROP | 21.94 | 4.64 |
30
+ | **Average**| **35.08** | **32.44** |
31
+
32
  **πŸš€ Motivations**
33
+ 1. To create a smaller, open-source, instruction-finetuned, ready-to-use model accessible for users with limited computational resources (lower-end consumer GPUs).
34
  2. To harness the strength of Falcon-RW-1B, a competitive model in its own right, and enhance its capabilities with instruction finetuning.
35
 
36
  ## πŸ“– How to Use
37
 
38
+ The model operates with a structured prompt format, incorporating `<SYS>`, `<INST>`, and `<RESP>` tags to demarcate different parts of the input. The system message and instruction are placed within these tags, with the `<RESP>` tag triggering the model's response.
39
 
40
  ### πŸ“ Example Code
41
 
 
48
 
49
  tokenizer = AutoTokenizer.from_pretrained(model)
50
  pipeline = transformers.pipeline(
51
+ 'text-generation',
52
  model=model,
53
  tokenizer=tokenizer,
54
  torch_dtype=torch.bfloat16,
55
+ device_map='auto',
56
  )
57
 
58
+ system_message = 'You are a helpful assistant. Give short answers.'
59
+ instruction = 'What is AI? Give some examples.'
60
+ prompt = f'<SYS> {system_message} <INST> {instruction} <RESP> '
61
 
62
  response = pipeline(
63
  prompt,
 
69
  # AI, or Artificial Intelligence, refers to the ability of machines and software to perform tasks that require human intelligence, such as learning, reasoning, and problem-solving. It can be used in various fields like computer science, engineering, medicine, and more. Some common applications include image recognition, speech translation, and natural language processing.
70
  ```
71
 
72
+ ## πŸ“¬ Contact
73
+ For further inquiries or feedback, please contact at eric.fu96@aol.com.