|
--- |
|
tags: |
|
- math |
|
- gsm8k |
|
- orchestration_of_experts |
|
datasets: |
|
- gsm8k |
|
--- |
|
|
|
# Leeroo Dedidcated Math Expert 🤗 |
|
|
|
The model is built by applying [Orchestration of Expert](https://arxiv.org/abs/2401.13979) for the math domain. It either generates solutions or, when necessary, utilizes GPT-4 to fill in gaps in its knowledge base. |
|
|
|
In evaluations using the GSM8k dataset of [OpenLLM Leaderboard](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard), Leeroo Math 7B model achieved an accuracy of 84.77% in 5-shot setting, positioning it among the top performers in its class and notably surpassing the MetaMath 7B model (its base model), which scores 68.84% on the same dataset. This was accomplished while relying on GPT-4 for responses to half of the questions posed by GSM8k. |
|
|
|
## Sample Usage |
|
|
|
```python |
|
from transformers import AutoModelForCausalLM, AutoTokenizer |
|
|
|
model = AutoModelForCausalLM.from_pretrained("leeroo/LeerooDedicated-Math", trust_remote_code=True) |
|
tokenizer = AutoTokenizer.from_pretrained("leeroo/LeerooDedicated-Math") |
|
device = model.device |
|
|
|
# the following question is answered by the leeroo expert |
|
question = "Natalia sold clips to 48 of her friends in April,and then she sold half as many clips in May.How many clips did Natalia sell altogether in April and May?" |
|
encodeds = tokenizer([question], return_tensors="pt") |
|
model_inputs = encodeds['input_ids'].to(device) |
|
generated_ids = model.generate(model_inputs, max_new_tokens=100, do_sample=False) |
|
decoded = tokenizer.batch_decode(generated_ids) |
|
print(decoded[0]) |
|
# Natalia sold 48 clips in April.\nIn May, she sold half as many clips as in April, |
|
# so she sold 48/2 = 24 clips.\nAltogether, Natalia sold 48 + 24 = 72 clips in April and May.\n#### 72\nThe answer is: 72</s> |
|
|
|
# sends the following questin to GPT4 |
|
question = "James loves to go swimming and has to swim across a 20-mile lake. He can swim at a pace of 2 miles per hour. He swims 60% of the distance. After that, he stops on an island and rests for half as long as the swimming time. He then finishes the remaining distance while going half the speed. How long did it take him to get across the lake?" |
|
encodeds = tokenizer([question], return_tensors="pt") |
|
model_inputs = encodeds['input_ids'].to(device) |
|
generated_ids = model.generate(model_inputs, max_new_tokens=100, do_sample=False) |
|
decoded = tokenizer.batch_decode(generated_ids) |
|
print(decoded[0]) |
|
# <GPT4></s> |
|
``` |
|
|
|
## Learn More |
|
|
|
🔍 To a deeper dive into our method and results, refer to [HF blog 🤗](https://huggingface.co/blog/alirezamsh/leeroo-multi-model-system), [publication](https://arxiv.org/abs/2401.13979), and [repository](https://github.com/Leeroo-AI/leeroo_orchestrator). |
|
🌍 Join Leeroo community for further updates: [Linkedin](https://www.linkedin.com/company/leeroo/?viewAsMember=true), [Discord](https://discord.gg/yp4PRwZj), [X](https://twitter.com/LeerooAI), [Website](https://www.leeroo.com/). |
|
|
|
## Citation |
|
``` |
|
@misc{mohammadshahi2024leeroo, |
|
title={Leeroo Orchestrator: Elevating LLMs Performance Through Model Integration}, |
|
author={Alireza Mohammadshahi and Ali Shaikh and Majid Yazdani}, |
|
year={2024}, |
|
eprint={2401.13979}, |
|
archivePrefix={arXiv}, |
|
primaryClass={cs.CL} |
|
} |
|
``` |
