README.md

---
license: other
license_name: qwen-research
license_link: https://huggingface.co/Qwen/Qwen2.5-3B/blob/main/LICENSE
language:
- en
pipeline_tag: text-generation
datasets:
- mlabonne/orpo-dpo-mix-40k
base_model:
- Qwen/Qwen2.5-3B
---

# Qwen2.5-3B-ORPO

This model is a finetuned model of the (Qwen2.5-3B base model](https://huggingface.co/Qwen/Qwen2.5-3B) by Qwen using ORPO over the [ORPO-DPO-mix dataset](https://huggingface.co/datasets/mlabonne/orpo-dpo-mix-40k) by M. Labonne.
We evaluate the model using several benchmarks working with an Eleuther evaluation harness. Apart from ensuring no reduction in general model performance, benchmarks testing for improved performance in logical and numerical reasoning
are applied. The results are inconclusive on most, but not all, metrics. This shows promise for further improvements on more tailored datasets or by further applying DPO for preference training.

## Model Details

### Model Description

Qwen2.5 is the latest series of Qwen large language models. We finetune the 3B model with the following specifications:

- **Developed by:** (Finetuned from) Qwen
- **Language(s) (NLP):** English
- **Finetuned from model:** Qwen2.5-3B
- **Model type:** Causal LM
- **Architecture:** Transformers with RoPE, SwiGLU, RMSNorm, Attention QKV bias and tied word embeddings
- **Number of Parameters:** 3.09B
- **Number of Paramaters (Non-Embedding):** 2.77B
- **Number of Layers:** 36
- **Number of Attention Heads (GQA):** 16 for Q and 2 for KV
- **Context Length:** Full 32,768 tokens

For additional details, we refer to the base model repository. 

### Model Sources

The Qwen2.5-3B base model can be found here:

- **Repository:** https://huggingface.co/Qwen/Qwen2.5-3B

## Uses

The model is finetuned but with only little performance increase over the base model in logical and numerical reasoning.
While better than the base model, we do not think it suffices for well-founded logical and numerical reasoning at this stage.
However, we detect no performance decrease for common sense natural reasoning.

### Direct Use

Common sense natural language reasoning.

### Downstream Use

Logical and numerical reasoning.

### Recommendations

Additional finetuning on different datasets, as well as preference training.

## Training Details

### Training Data

We use the [ORPO-DPO-mix dataset](https://huggingface.co/datasets/mlabonne/orpo-dpo-mix-40k) by M. Labonne.
It is a dataset is designed for ORPO or DPO training. See Fine-tune Llama 3 with ORPO for more information about how to use it.

### Training Procedure

We used the trl [ORPO trainer](https://huggingface.co/docs/trl/main/en/orpo_trainer) for finetuning over four epochs with batch size two.
Moreover, we used [LoRa](https://arxiv.org/abs/2106.09685) for parameter efficient training by targeting only particular parts of the base model architecture.

### Training Hyperparameters

- **Training regime:** fp16 non-mixed precision
- **Max lenght:** 4096
- **Max prompt length:** 4096
- **Batch size:** 2
- **Epochs trained:** 4
- **Modules targeted:** q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
- **Bias:** None

All remaining hyperparameters were kept standard.

# Evaluation

We evaluate base and finetuned models on four general benchmarks and two usecase specific one. We work with an Eleuther test harness.
Our usecase is logical and numerical reasoning.

## Benchmarks used:

1. GENERAL A: Commonsense natural language reasoning.
  1.1 HellaSwag
  1.2 WinoGrande

2. GENERAL B: Robustness and grammatical understanding.
  2.1 ANLI
  2.2 BLiMP

3. USECASE: Logical and numerical reasoning.
  3.1 Arithmetic
  3.2 ASDiv

## Summary of results:

Within standard error, there is no difference between base and finetuned model on any general benchmark. This suggests there was no drop in performance for the chosen tasks due to finetuning.
Benchmarks for logical and numerical reasoning are more mixed. Without standard error, the finetuned model generally outperforms the base model. However, this lies - often just about - within standard error.
The finetuned model *does* outperform the base model even accounting for standard error with maximal conservative bias on [**arithmetic_5da**](https://arxiv.org/abs/2005.14165).
This is of interest, since this benchmarks a model's ability to add five digits, and addition is *the* fundamental arithmetic operation. Subtraction appears generally harder for both the finetuned and base models, even as the finetuned model performs better.
We highlight the relevant rows for five-digit addition and subtraction for easy comparison. Moreover, we give a visualisation of the performance gain *without standard error* for all usecase specific benchmarks at the end of the model card.

## Evaluation results:

**BASE**

<figure>
  
|  Tasks   | Version | Filter | n-shot | Metric |   | Value |   | Stderr |
|----------|--------:|--------|-------:|--------|---|------:|---|-------:|
|hellaswag |       1 | none   |      0 |acc     |↑  | 0.5492|±  |  0.0050|
|          |         | none   |      0 |acc_norm|↑  | 0.7353|±  |  0.0044|
|winogrande|       1 | none   |      0 |acc     |↑  | 0.6851|±  |  0.0131|
|anli_r1   |       1 | none   |      0 |acc     |↑  | 0.4670|±  |  0.0158|
|anli_r2   |       1 | none   |      0 |acc     |↑  | 0.4440|±  |  0.0157|
|anli_r3   |       1 | none   |      0 |acc     |↑  | 0.4467|±  |  0.0144|
|blimp     |       2 | none   |      0 |acc     |↑  | 0.7250|±  |  0.0016|
<figcaption>Collect GENERAL benchmarks results for the base model.</figcaption>
</figure>

<figure>
  
|  Tasks       | Version | Filter | n-shot | Metric |   | Value |   | Stderr |
|--------------|--------:|--------|-------:|--------|---|------:|---|-------:|
|arithmetic_1dc|       1 | none   |      0 |acc     |↑  | 0.6780|±  |  0.0105|
|arithmetic_2da|       1 | none   |      0 |acc     |↑  | 0.0095|±  |  0.0022|
|arithmetic_2dm|       1 | none   |      0 |acc     |↑  | 0.0230|±  |  0.0034|
|arithmetic_2ds|       1 | none   |      0 |acc     |↑  | 0.0280|±  |  0.0037|
|arithmetic_3da|       1 | none   |      0 |acc     |↑  | 0.0075|±  |  0.0019|
|arithmetic_3ds|       1 | none   |      0 |acc     |↑  | 0.0055|±  |  0.0017|
|arithmetic_4da|       1 | none   |      0 |acc     |↑  | 0.0675|±  |  0.0056|
|arithmetic_4ds|       1 | none   |      0 |acc     |↑  | 0.0010|±  |  0.0007|
|**arithmetic_5da**|       1 | none   |      0 |acc     |↑  | **0.3720**|±  |  **0.0108**|
|*arithmetic_5ds*|       1 | none   |      0 |acc     |↑  | *0.0260*|±  |  *0.0036*|
|asdiv         |       1 | none   |      0 |acc     |↑  | 0.0187|±  |  0.0028|
<figcaption>Collected USECASE benchmarks results for the base model.</figcaption>

</figure>

**FINETUNED**

<figure>

|  Tasks   | Version | Filter | n-shot | Metric |   | Value |   | Stderr |
|----------|--------:|--------|-------:|--------|---|------:|---|-------:|
|hellaswag |       1 | none   |      0 |acc     |↑  | 0.5490|±  |  0.0050|
|          |         | none   |      0 |acc_norm|↑  | 0.7358|±  |  0.0044|
|winogrande|       1 | none   |      0 |acc     |↑  | 0.6827|±  |  0.0131|
|anli_r1   |       1 | none   |      0 |acc     |↑  | 0.4660|±  |  0.0158|
|anli_r2   |       1 | none   |      0 |acc     |↑  | 0.4380|±  |  0.0157|
|anli_r3   |       1 | none   |      0 |acc     |↑  | 0.4408|±  |  0.0143|
|blimp     |       2 | none   |      0 |acc     |↑  | 0.7253|±  |  0.0016|
<figcaption>Collect GENERAL benchmarks results for the finetuned model.</figcaption>
</figure>

<figure>
  
|  Tasks       | Version | Filter | n-shot | Metric |   | Value |   | Stderr |
|--------------|--------:|--------|-------:|--------|---|------:|---|-------:|
|arithmetic_1dc|       1 | none   |      0 |acc     |↑  | 0.6725|±  |  0.0105|
|arithmetic_2da|       1 | none   |      0 |acc     |↑  | 0.0095|±  |  0.0022|
|arithmetic_2dm|       1 | none   |      0 |acc     |↑  | 0.0230|±  |  0.0034|
|arithmetic_2ds|       1 | none   |      0 |acc     |↑  | 0.0280|±  |  0.0037|
|arithmetic_3da|       1 | none   |      0 |acc     |↑  | 0.0080|±  |  0.0020|
|arithmetic_3ds|       1 | none   |      0 |acc     |↑  | 0.0055|±  |  0.0017|
|arithmetic_4da|       1 | none   |      0 |acc     |↑  | 0.0710|±  |  0.0057|
|arithmetic_4ds|       1 | none   |      0 |acc     |↑  | 0.0005|±  |  0.0005|
|**arithmetic_5da**|       1 | none   |      0 |acc     |↑  | **0.4005**|±  |  **0.0110**|
|*arithmetic_5ds*|       1 | none   |      0 |acc     |↑  | *0.0285*|±  |  *0.0037*|
|asdiv         |       1 | none   |      0 |acc     |↑  | 0.0204|±  |  0.0029|
<figcaption>Collected USECASE benchmarks results for the finetuned model.</figcaption>

</figure>

**VISUALISATION OF USECASE BENCHMARK RESULTS**

USECASE benchmark results given as percentage change of finetuned model relative to base model. **Does not account for standard error.**
For visualisation purposes we drop "arithmetic_4ds" since results for both models are very small and dominated by standard error.

![USECASE benchmark results](./eval_results.png)