katanemo/Arch-Router-test-v1

router--Qwen

This model is a fine-tuned version of Qwen/Qwen2.5-1.5B-Instruct on the router_multi and the router_single datasets. It achieves the following results on the evaluation set:

• Loss: 0.0410
• Num Input Tokens Seen: 28301728

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 5e-05
• train_batch_size: 2
• eval_batch_size: 2
• seed: 42
• gradient_accumulation_steps: 8
• total_train_batch_size: 16
• optimizer: Use adamw_torch with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: cosine
• num_epochs: 2.0

Training results

Training Loss	Epoch	Step	Validation Loss	Input Tokens Seen
0.1164	0.2833	200	0.1030	3969376
0.0821	0.5666	400	0.0944	7980480
0.1162	0.8499	600	0.0783	12021568
0.0613	1.1331	800	0.0695	16010784
0.0392	1.4164	1000	0.0553	20056096
0.0426	1.6997	1200	0.0459	24059776
0.0367	1.9830	1400	0.0410	28051872

Framework versions

• Transformers 4.49.0
• Pytorch 2.6.0+cu124
• Datasets 3.2.0
• Tokenizers 0.21.0