---
base_model: gpt2
library_name: Distily
license: mit
tags:
- generated_from_trainer
model-index:
- name: distily_bench_gpt2_simple_objectives2
  results: []
---

# distily_bench_gpt2_simple_objectives2

This student model is distilled from the teacher model [gpt2](https://huggingface.co/gpt2) using the dataset (unspecified).

The [Distily](https://github.com/lapp0/distily) library was used for this distillation.

It achieves the following results on the evaluation set:
- eval_enwikippl: 26585.3379
- eval_frwikippl: 34195.0625
- eval_zhwikippl: 50038.4062
- eval_loss: 0.0690
- eval_runtime: 32.5862
- eval_samples_per_second: 61.376
- eval_steps_per_second: 7.672

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment.

## Model description

More information needed

## Intended uses & limitations

More information needed

## Training and evaluation data

More information needed
-->

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- distillation_objective: MultiObjective(logits_weight=1, logits_loss_fn=(fn:cosine_distance_loss()), activations_weight=0.2, activations_loss_fn=(fn:soft_mse_loss()), attentions_weight=0, attentions_loss_fn=(fn:soft_mse_loss()))
- train_embeddings: True
- learning_rate: 4e-05
- train_batch_size: 8
- eval_batch_size: 8
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: constant
- num_epochs: 1.0

### Resource Usage
Peak GPU Memory: 10.3934 GB

### Eval-Phase Metrics
| step | epoch | enwikippl | frwikippl | loss | runtime | samples_per_second | steps_per_second | zhwikippl |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| **teacher eval** |  | 30.2086 | 57.2728 |  |  |  |  | 18.1784 |
| 0 | 0 | 56797.875 | 58468.6992 | 1.0 | 32.7055 | 61.152 | 7.644 | 59002.2891 |
| 1000 | 0.0404 | 27961.6797 | 33253.5469 | 0.0687 | 32.6144 | 61.323 | 7.665 | 48954.4688 |
| 2000 | 0.0808 | 28987.5293 | 34209.5430 | 0.0692 | 32.5898 | 61.369 | 7.671 | 49453.8398 |
| 3000 | 0.1212 | 31211.6582 | 36006.2383 | 0.0684 | 32.5399 | 61.463 | 7.683 | 56097.9492 |
| 4000 | 0.1616 | 30200.7598 | 35859.2969 | 0.0683 | 32.7214 | 61.122 | 7.64 | 55147.2305 |
| 5000 | 0.2020 | 29574.1465 | 34793.3281 | 0.0687 | 32.5565 | 61.432 | 7.679 | 52868.5352 |
| 6000 | 0.2424 | 28607.3867 | 34403.0312 | 0.0681 | 32.4239 | 61.683 | 7.71 | 52629.0312 |
| 7000 | 0.2828 | 25131.9219 | 33173.9180 | 0.0687 | 32.5743 | 61.398 | 7.675 | 52278.8164 |
| 8000 | 0.3232 | 27496.5820 | 34228.8320 | 0.0682 | 32.6299 | 61.293 | 7.662 | 52896.7773 |
| 9000 | 0.3636 | 26585.3379 | 34195.0625 | 0.0690 | 32.5862 | 61.376 | 7.672 | 50038.4062 |
| 10000 | 0.4040 | 27479.4902 | 33309.8555 | 0.0692 | 32.6173 | 61.317 | 7.665 | 53123.3086 |
| 11000 | 0.4444 | 29032.5703 | 32465.8555 | 0.0687 | 32.5477 | 61.448 | 7.681 | 54270.5625 |
| 12000 | 0.4848 | 25965.1055 | 33366.2578 | 0.0679 | 32.6589 | 61.239 | 7.655 | 56639.9258 |
| 13000 | 0.5253 | 28022.5176 | 34031.5195 | 0.0687 | 32.7256 | 61.114 | 7.639 | 55873.6484 |
| 14000 | 0.5657 | 29721.4824 | 33602.3359 | 0.0682 | 32.4915 | 61.555 | 7.694 | 54241.5859 |
| 15000 | 0.6061 | 23164.0742 | 32511.6797 | 0.0678 | 32.6017 | 61.347 | 7.668 | 52727.5469 |
| 16000 | 0.6465 | 22154.2578 | 34514.7969 | 0.0686 | 32.5602 | 61.425 | 7.678 | 55412.9883 |
| 17000 | 0.6869 | 28816.9688 | 36942.2734 | 0.0689 | 32.582 | 61.384 | 7.673 | 54677.9062 |
| 18000 | 0.7273 | 30663.9199 | 36817.4492 | 0.0686 | 32.6509 | 61.254 | 7.657 | 52769.8047 |
| 19000 | 0.7677 | 31843.2832 | 37875.9023 | 0.0682 | 32.5752 | 61.396 | 7.675 | 53094.9453 |
| 20000 | 0.8081 | 26705.3535 | 35377.125 | 0.0677 | 32.5515 | 61.441 | 7.68 | 55873.6484 |
| 21000 | 0.8485 | 31013.5449 | 35662.6172 | 0.0671 | 32.8968 | 60.796 | 7.6 | 53579.2930 |
| 22000 | 0.8889 | 31917.5293 | 33950.0234 | 0.0672 | 32.7146 | 61.135 | 7.642 | 53779.9688 |
| 23000 | 0.9293 | 30907.7520 | 34783.5430 | 0.0678 | 32.8995 | 60.791 | 7.599 | 53564.9883 |
| 24000 | 0.9697 | 30893.3711 | 34383.6484 | 0.0676 | 32.7142 | 61.135 | 7.642 | 54039.1367 |
| 24750 | 1.0 | 37617.5469 | 41219.9453 | 0.0685 | 32.7176 | 61.129 | 7.641 | 52572.8477 |

### Framework versions
- Distily 0.2.0
- Transformers 4.44.0
- Pytorch 2.3.0
- Datasets 2.20.0