Nemo
Collection
Mistral Nemo 12B finetunes and merges • 6 items • Updated • 2
How to use from
SGLangUse Docker images
docker run --gpus all \
--shm-size 32g \
-p 30000:30000 \
-v ~/.cache/huggingface:/root/.cache/huggingface \
--env "HF_TOKEN=<secret>" \
--ipc=host \
lmsysorg/sglang:latest \
python3 -m sglang.launch_server \
--model-path "nbeerbower/Lyra-Gutenberg-mistral-nemo-12B" \
--host 0.0.0.0 \
--port 30000# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
-H "Content-Type: application/json" \
--data '{
"model": "nbeerbower/Lyra-Gutenberg-mistral-nemo-12B",
"prompt": "Once upon a time,",
"max_tokens": 512,
"temperature": 0.5
}'Quick Links
Lyra-Gutenberg-12B
Sao10K/MN-12B-Lyra-v1 finetuned on jondurbin/gutenberg-dpo-v0.1.
Method
Finetuned using an A100 on Google Colab for 3 epochs.
Open LLM Leaderboard Evaluation Results
Detailed results can be found here
| Metric | Value |
|---|---|
| Avg. | 22.57 |
| IFEval (0-Shot) | 34.95 |
| BBH (3-Shot) | 36.99 |
| MATH Lvl 5 (4-Shot) | 8.31 |
| GPQA (0-shot) | 11.19 |
| MuSR (0-shot) | 14.76 |
| MMLU-PRO (5-shot) | 29.20 |
- Downloads last month
- 53
Model tree for nbeerbower/Lyra-Gutenberg-mistral-nemo-12B
Dataset used to train nbeerbower/Lyra-Gutenberg-mistral-nemo-12B
Spaces using nbeerbower/Lyra-Gutenberg-mistral-nemo-12B 7
Collection including nbeerbower/Lyra-Gutenberg-mistral-nemo-12B
Evaluation results
- strict accuracy on IFEval (0-Shot)Open LLM Leaderboard34.950
- normalized accuracy on BBH (3-Shot)Open LLM Leaderboard36.990
- exact match on MATH Lvl 5 (4-Shot)Open LLM Leaderboard8.310
- acc_norm on GPQA (0-shot)Open LLM Leaderboard11.190
- acc_norm on MuSR (0-shot)Open LLM Leaderboard14.760
- accuracy on MMLU-PRO (5-shot)test set Open LLM Leaderboard29.200
Install from pip and serve model
# Install SGLang from pip: pip install sglang# Start the SGLang server: python3 -m sglang.launch_server \ --model-path "nbeerbower/Lyra-Gutenberg-mistral-nemo-12B" \ --host 0.0.0.0 \ --port 30000# Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "nbeerbower/Lyra-Gutenberg-mistral-nemo-12B", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }'