Hugging Face's logo

tokyotech-llm
/
Llama-3.1-Swallow-70B-Instruct-v0.1

Text Generation
Transformers
Safetensors
English
Japanese
llama
conversational
text-generation-inference
Inference Endpoints
Model card Files Community
1
Taishi-N324 commited on
Commit
bfc3c83
·
verified ·
1 Parent(s): 2e53320

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +2 -9
README.md CHANGED
@@ -51,21 +51,14 @@ The website [https://swallow-llm.github.io/](https://swallow-llm.github.io/) pro
51
  |---|---|---|---|---|---|---|---|---|---|---|---|
52
  | |4-shot|4-shot|4-shot|4-shot|1-shot|4-shot|4-shot|4-shot|5-shot|0-shot| |
53
  | | |EM acc|Char-F1|Char-F1|Char-F1|ROUGE-2|EM acc|BLEU|BLEU|EM acc|pass@1| |
54
- | Gemma 2 27B IT | 0.9562 | 0.5413 | 0.5755 | 0.8832 | 0.1648 | 0.7000 | 0.2900 | 0.2500 | 0.6701 | 0.6293 | 0.5660 |
55
- | Phi-3.5-MoE Instruct | 0.9321 | 0.4416 | 0.4920 | 0.9079 | 0.2255 | 0.7120 | 0.2575 | 0.2024 | 0.6447 | 0.4213 | 0.5237 |
56
- | GRIN-MoE | 0.8606 | 0.4622 | 0.3943 | 0.8877 | 0.0302 | 0.6400 | 0.2300 | 0.1911 | 0.5696 | 0.4476 | 0.4713 |
57
- | KARAKURI LM 70B Chat v0.1 | 0.8847 | 0.5139 | 0.5668 | 0.9096 | 0.1369 | 0.2800 | 0.2526 | 0.2095 | 0.4648 | 0.2354 | 0.4454 |
58
- | Swallow-70b-instruct-v0.1 | 0.9231 | 0.5654 | 0.5751 | 0.9036 | 0.1861 | 0.4160 | 0.2619 | 0.2318 | 0.5727 | 0.2835 | 0.4919 |
59
  | Llama 3 70B Instruct | 0.9419 | 0.6114 | 0.5506 | 0.9164 | 0.1912 | 0.7200 | 0.2708 | 0.2350 | 0.6789 | 0.6610 | 0.5777 |
60
  | Llama 3.1 70B Instruct | 0.9482 | 0.6246 | 0.5781 | 0.9201 | 0.1772 | 0.7440 | 0.2805 | 0.2472 | 0.7323 | 0.6933 | 0.5945 |
61
  | Llama 3 Youko 70B Instruct | 0.9526 | 0.6252 | 0.5853 | 0.9215 | 0.1983 | 0.7400 | 0.2633 | 0.2245 | 0.7170 | 0.6098 | 0.5838 |
62
- | Llama-3.1-70B-Japanese-Instruct-2407 | 0.9562 | 0.6466 | 0.6602 | 0.9187 | 0.1564 | 0.7480 | 0.2901 | 0.2410 | 0.7227 | 0.6274 | 0.5967 |
63
  | Llama 3 heron brain 70B v0.3 | 0.9660 | 0.6643 | 0.6817 | 0.9221 | 0.2611 | 0.7720 | 0.3093 | 0.2578 | 0.7077 | 0.6079 | 0.6150 |
64
  | Llama 3 Swallow 70B Instruct | 0.9607 | 0.6188 | 0.6026 | 0.9236 | 0.1389 | 0.6560 | 0.2724 | 0.2532 | 0.6572 | 0.6000 | 0.5683 |
65
  | Llama 3.1 Swallow 70B Instruct | 0.9598 | 0.6192 | 0.6605 | 0.9235 | 0.1938 | 0.7760 | 0.3123 | 0.2593 | 0.7117 | 0.4713 | 0.5887 |
66
- | Qwen2-72B-Instruct | 0.9634 | 0.6268 | 0.5418 | 0.9210 | 0.1644 | 0.7840 | 0.2592 | 0.2327 | 0.7713 | 0.6909 | 0.5955 |
67
- | Qwen2.5-72B-Instruct | 0.9696 | 0.5699 | 0.5811 | 0.7381 | 0.1706 | 0.8360 | 0.2269 | 0.2179 | 0.7899 | 0.6256 | 0.5726 |
68
- | Mixtral-8x22B-Instruct-v0.1 | 0.9053 | 0.5001 | 0.4609 | 0.9186 | 0.2060 | 0.6760 | 0.2327 | 0.2313 | 0.6094 | 0.5787 | 0.5319 |
69
 
70
  ### English tasks
71
 
 
51
  |---|---|---|---|---|---|---|---|---|---|---|---|
52
  | |4-shot|4-shot|4-shot|4-shot|1-shot|4-shot|4-shot|4-shot|5-shot|0-shot| |
53
  | | |EM acc|Char-F1|Char-F1|Char-F1|ROUGE-2|EM acc|BLEU|BLEU|EM acc|pass@1| |
54
+ | Qwen2-72B-Instruct | 0.9634 | 0.6268 | 0.5418 | 0.9210 | 0.1644 | 0.7840 | 0.2592 | 0.2327 | 0.7713 | 0.6909 | 0.5955 |
55
+ | Qwen2.5-72B-Instruct | 0.9696 | 0.5699 | 0.5811 | 0.7381 | 0.1706 | 0.8360 | 0.2269 | 0.2179 | 0.7899 | 0.6256 | 0.5726 |
 
 
 
56
  | Llama 3 70B Instruct | 0.9419 | 0.6114 | 0.5506 | 0.9164 | 0.1912 | 0.7200 | 0.2708 | 0.2350 | 0.6789 | 0.6610 | 0.5777 |
57
  | Llama 3.1 70B Instruct | 0.9482 | 0.6246 | 0.5781 | 0.9201 | 0.1772 | 0.7440 | 0.2805 | 0.2472 | 0.7323 | 0.6933 | 0.5945 |
58
  | Llama 3 Youko 70B Instruct | 0.9526 | 0.6252 | 0.5853 | 0.9215 | 0.1983 | 0.7400 | 0.2633 | 0.2245 | 0.7170 | 0.6098 | 0.5838 |
 
59
  | Llama 3 heron brain 70B v0.3 | 0.9660 | 0.6643 | 0.6817 | 0.9221 | 0.2611 | 0.7720 | 0.3093 | 0.2578 | 0.7077 | 0.6079 | 0.6150 |
60
  | Llama 3 Swallow 70B Instruct | 0.9607 | 0.6188 | 0.6026 | 0.9236 | 0.1389 | 0.6560 | 0.2724 | 0.2532 | 0.6572 | 0.6000 | 0.5683 |
61
  | Llama 3.1 Swallow 70B Instruct | 0.9598 | 0.6192 | 0.6605 | 0.9235 | 0.1938 | 0.7760 | 0.3123 | 0.2593 | 0.7117 | 0.4713 | 0.5887 |
 
 
 
62
 
63
  ### English tasks
64