Coder1.8-ORPO-TEST
Model Description
Test model for ORPO finetune method, trained on ~20k code examples for 1 epoch on 2 x A40 cards with 4-bit QLora (lora rank=lora alpha=16).
Disclaimer
This is a test model and may generate incorrect responses. Use at your own risk.
Train Details
- Base: Qwen1.5-1.8B
- Training Data: ~20k code examples
- Epochs: 1
- Method: ORPO
- Hardware: 2 x A40
- Quantization: 4-bit QLora
- Lora Rank/Alpha: 16
Limitations
Limited training data and quantization may impact performance.
Join the Discussion
Have questions or feedback? Join our Discord server Here.
Open LLM Leaderboard Evaluation Results
Detailed results can be found here
| Metric | Value |
|---|---|
| Avg. | 45.76 |
| AI2 Reasoning Challenge (25-Shot) | 38.82 |
| HellaSwag (10-Shot) | 60.48 |
| MMLU (5-Shot) | 46.70 |
| TruthfulQA (0-shot) | 41.38 |
| Winogrande (5-shot) | 59.75 |
| GSM8k (5-shot) | 27.45 |
- Downloads last month
- 1,284
Dataset used to train raincandy-u/Coder1.8-ORPO-TEST
Evaluation results
- normalized accuracy on AI2 Reasoning Challenge (25-Shot)test set Open LLM Leaderboard38.820
- normalized accuracy on HellaSwag (10-Shot)validation set Open LLM Leaderboard60.480
- accuracy on MMLU (5-Shot)test set Open LLM Leaderboard46.700
- mc2 on TruthfulQA (0-shot)validation set Open LLM Leaderboard41.380
- accuracy on Winogrande (5-shot)validation set Open LLM Leaderboard59.750
- accuracy on GSM8k (5-shot)test set Open LLM Leaderboard27.450