tensorplex-labs
/

Sumo-T9-7B-v0.1

@@ -11,7 +11,6 @@ tags:
 - base-model
 - bittensor
 - decentralized AI
-- Web3
 datasets:
 - tiiuae/falcon-refinedweb
 ---
@@ -44,18 +43,17 @@ Since the parameter limit was upgraded to 7 billion on April 19, 2024, Tensorple
 - **Architecture**: Adopted Llama-style architecture with 6.9 billion parameters
 - **Training Data**: Trained on the tiiuae/falcon-refinedweb dataset
 - **Training Objective**: Causal Language Modeling (next token prediction)
 Sumo-Qyuu-7B-v0.1 features a larger vocabulary size (100k), compatible with the GPT-4 tokenizer, ensuring its versatility across various natural language processing tasks.
 ### Model Sources
 - **Bittensor Subnet9 Leaderboard:** [https://huggingface.co/spaces/RaoFoundation/pretraining-leaderboard](https://huggingface.co/spaces/RaoFoundation/pretraining-leaderboard)
 - **Bittensor Subnet9 Repository:** [https://github.com/RaoFoundation/pretraining/tree/main](https://github.com/RaoFoundation/pretraining/tree/main)
-## Usage
-⛔ **This is a pretrained base model, which hasn't been aligned yet. Use with caution or finetune further on downstream tasks before deployment.**
 ## How to Get Started with the Model
 Use the code below to get started with the model.
@@ -93,26 +91,23 @@ for seq in sequences:
 ### Training Data
-This model has been trained with [tiiuae/falcon-refinedweb](https://huggingface.co/datasets/tiiuae/falcon-refinedweb) dataset continuously.
 ## Evaluation
 Sumo-Qyuu-7B-v0.1 has outperformed notable models such as TII Falcon 7B, Meta's Llama-2-7b and Llama-1-7b in zero-shot performance,
 establishing itself as the leading model in aggregate across various evaluation tasks.
-Such benchmarks include ARC Challenge, GSM8K, HellaSwag, MMLU, TruthfulQA MC2, and Winogrande.
-|                                  |      tensorplex-labs/Sumo-Qyuu-7B-v0.1 |   NousResearch/Llama-2-7b-hf |   yahma/llama-7b-hf |   tiiuae/falcon-7b |
-|----------------------------------|----------------------------------------|------------------------------|---------------------|--------------------|
-| **avg**                          |                            **47.85**   |                      47.31   |               44.22 |             42.03  |
-| arc_challenge (acc_norm, 0-shot) |                                47.53   |                      46.16   |               44.88 |             43.43  |
-| gsm8k (exact_match, 5-shot)      |                                10.46   |                      13.27   |               10.39 |             05.23  |
-| hellaswag (acc_norm, 0-shot)     |                                76.66   |                      75.97   |               76.19 |             76.33  |
-| mmlu (acc, 0-shot)               |                                44.26   |                      40.78   |               29.68 |             25.72  |
-| truthfulqa_mc2 (acc, 0-shot)     |                                37.29   |                      39.00   |               34.01 |             34.27  |
-| winogrande (acc, 0-shot)         |                                70.88   |                      68.67   |               70.17 |             67.17  |
-[LM Evaluation Harness Repository](https://github.com/EleutherAI/lm-evaluation-harness)
 ## Future Plans

 - base-model
 - bittensor
 - decentralized AI
 datasets:
 - tiiuae/falcon-refinedweb
 ---
 - **Architecture**: Adopted Llama-style architecture with 6.9 billion parameters
 - **Training Data**: Trained on the tiiuae/falcon-refinedweb dataset
 - **Training Objective**: Causal Language Modeling (next token prediction)
+- **Original Model Repo**: [tensorplex-labs/pretraining-sn9-7B-1](https://huggingface.co/tensorplex-labs/pretraining-sn9-7B-1)
 Sumo-Qyuu-7B-v0.1 features a larger vocabulary size (100k), compatible with the GPT-4 tokenizer, ensuring its versatility across various natural language processing tasks.
+⛔ **This is a pretrained base model, which hasn't been aligned yet. Use with caution or finetune further on downstream tasks before deployment.**
 ### Model Sources
 - **Bittensor Subnet9 Leaderboard:** [https://huggingface.co/spaces/RaoFoundation/pretraining-leaderboard](https://huggingface.co/spaces/RaoFoundation/pretraining-leaderboard)
 - **Bittensor Subnet9 Repository:** [https://github.com/RaoFoundation/pretraining/tree/main](https://github.com/RaoFoundation/pretraining/tree/main)
 ## How to Get Started with the Model
 Use the code below to get started with the model.
 ### Training Data
+This model has been trained with [tiiuae/falcon-refinedweb](https://huggingface.co/datasets/tiiuae/falcon-refinedweb) dataset, and still ongoing continuously.
 ## Evaluation
 Sumo-Qyuu-7B-v0.1 has outperformed notable models such as TII Falcon 7B, Meta's Llama-2-7b and Llama-1-7b in zero-shot performance,
 establishing itself as the leading model in aggregate across various evaluation tasks.
+Such benchmarks include ARC Challenge, GSM8K, HellaSwag, MMLU, TruthfulQA, and Winogrande.
+|                                       |    avg     |   arc_challenge |   gsm8k |   hellaswag |   mmlu |   truthfulqa_mc2 |   winogrande |
+|:--------------------------------------|-----------:|----------------:|--------:|------------:|-------:|-----------------:|-------------:|
+| meta-llama/Meta-Llama-3-8B            | 0.6009     |          0.5333 |  0.4913 |      0.7906 | 0.621  |           0.4392 |       0.7301 |
+| **tensorplex-labs/Sumo-Qyuu-7B-v0.1** | **0.4769** |          0.4753 |  0.1031 |      0.7666 | 0.4426 |           0.3723 |       0.7017 |
+| meta-llama/Llama-2-7b-hf              | 0.473      |          0.4625 |  0.1213 |      0.7597 | 0.4123 |           0.3896 |       0.693  |
+| huggyllama/llama-7b                   | 0.4386     |          0.4471 |  0.0849 |      0.7621 | 0.2973 |           0.3408 |       0.6993 |
+| tiiuae/falcon-7b                      | 0.4189     |          0.4343 |  0.0432 |      0.7636 | 0.2582 |           0.3428 |       0.6717 |
 ## Future Plans