---
library_name: peft
tags:
- generated_from_trainer
base_model: mahdibaghbanzadeh/seqsight_4096_512_27M
metrics:
- accuracy
model-index:
- name: GUE_EMP_H4-seqsight_4096_512_27M-L8_f
  results: []
---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

# GUE_EMP_H4-seqsight_4096_512_27M-L8_f

This model is a fine-tuned version of [mahdibaghbanzadeh/seqsight_4096_512_27M](https://huggingface.co/mahdibaghbanzadeh/seqsight_4096_512_27M) on the [mahdibaghbanzadeh/GUE_EMP_H4](https://huggingface.co/datasets/mahdibaghbanzadeh/GUE_EMP_H4) dataset.
It achieves the following results on the evaluation set:
- Loss: 0.2553
- F1 Score: 0.9075
- Accuracy: 0.9076

## Model description

More information needed

## Intended uses & limitations

More information needed

## Training and evaluation data

More information needed

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 0.0005
- train_batch_size: 128
- eval_batch_size: 128
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- training_steps: 10000

### Training results

| Training Loss | Epoch  | Step  | Validation Loss | F1 Score | Accuracy |
|:-------------:|:------:|:-----:|:---------------:|:--------:|:--------:|
| 0.3464        | 2.17   | 200   | 0.2854          | 0.8926   | 0.8925   |
| 0.2702        | 4.35   | 400   | 0.2757          | 0.8975   | 0.8973   |
| 0.2566        | 6.52   | 600   | 0.2777          | 0.8946   | 0.8946   |
| 0.2531        | 8.7    | 800   | 0.2828          | 0.8982   | 0.8980   |
| 0.2364        | 10.87  | 1000  | 0.2706          | 0.8973   | 0.8973   |
| 0.2326        | 13.04  | 1200  | 0.2738          | 0.9000   | 0.9001   |
| 0.2221        | 15.22  | 1400  | 0.2717          | 0.9077   | 0.9076   |
| 0.2124        | 17.39  | 1600  | 0.2961          | 0.8929   | 0.8925   |
| 0.2089        | 19.57  | 1800  | 0.2771          | 0.8959   | 0.8960   |
| 0.2005        | 21.74  | 2000  | 0.2934          | 0.8962   | 0.8960   |
| 0.1924        | 23.91  | 2200  | 0.2988          | 0.8959   | 0.8960   |
| 0.1896        | 26.09  | 2400  | 0.2897          | 0.8912   | 0.8912   |
| 0.183         | 28.26  | 2600  | 0.3030          | 0.8896   | 0.8898   |
| 0.1757        | 30.43  | 2800  | 0.3046          | 0.8877   | 0.8877   |
| 0.1693        | 32.61  | 3000  | 0.3091          | 0.8933   | 0.8932   |
| 0.1624        | 34.78  | 3200  | 0.3127          | 0.8852   | 0.8850   |
| 0.1625        | 36.96  | 3400  | 0.3129          | 0.8920   | 0.8919   |
| 0.1544        | 39.13  | 3600  | 0.3324          | 0.8822   | 0.8823   |
| 0.1483        | 41.3   | 3800  | 0.3317          | 0.8889   | 0.8891   |
| 0.1473        | 43.48  | 4000  | 0.3315          | 0.8836   | 0.8836   |
| 0.1454        | 45.65  | 4200  | 0.3341          | 0.8840   | 0.8843   |
| 0.1392        | 47.83  | 4400  | 0.3500          | 0.8776   | 0.8775   |
| 0.1348        | 50.0   | 4600  | 0.3604          | 0.8771   | 0.8775   |
| 0.1301        | 52.17  | 4800  | 0.3675          | 0.8794   | 0.8795   |
| 0.1285        | 54.35  | 5000  | 0.3700          | 0.8751   | 0.8754   |
| 0.1233        | 56.52  | 5200  | 0.3709          | 0.8859   | 0.8857   |
| 0.1243        | 58.7   | 5400  | 0.3766          | 0.8738   | 0.8741   |
| 0.1199        | 60.87  | 5600  | 0.3872          | 0.8817   | 0.8816   |
| 0.1162        | 63.04  | 5800  | 0.3914          | 0.8795   | 0.8795   |
| 0.1153        | 65.22  | 6000  | 0.3962          | 0.8736   | 0.8741   |
| 0.1063        | 67.39  | 6200  | 0.3987          | 0.8748   | 0.8747   |
| 0.1061        | 69.57  | 6400  | 0.4121          | 0.8685   | 0.8686   |
| 0.1072        | 71.74  | 6600  | 0.4133          | 0.8754   | 0.8754   |
| 0.1044        | 73.91  | 6800  | 0.4176          | 0.8774   | 0.8775   |
| 0.1001        | 76.09  | 7000  | 0.4241          | 0.8772   | 0.8775   |
| 0.1025        | 78.26  | 7200  | 0.4178          | 0.8717   | 0.8720   |
| 0.0978        | 80.43  | 7400  | 0.4276          | 0.8725   | 0.8727   |
| 0.0962        | 82.61  | 7600  | 0.4393          | 0.8707   | 0.8713   |
| 0.0963        | 84.78  | 7800  | 0.4390          | 0.8787   | 0.8789   |
| 0.0934        | 86.96  | 8000  | 0.4465          | 0.8703   | 0.8706   |
| 0.0917        | 89.13  | 8200  | 0.4537          | 0.8696   | 0.8700   |
| 0.0902        | 91.3   | 8400  | 0.4595          | 0.8702   | 0.8706   |
| 0.0857        | 93.48  | 8600  | 0.4673          | 0.8737   | 0.8741   |
| 0.0884        | 95.65  | 8800  | 0.4660          | 0.8701   | 0.8706   |
| 0.0851        | 97.83  | 9000  | 0.4629          | 0.8689   | 0.8693   |
| 0.0847        | 100.0  | 9200  | 0.4669          | 0.8691   | 0.8693   |
| 0.0842        | 102.17 | 9400  | 0.4653          | 0.8704   | 0.8706   |
| 0.083         | 104.35 | 9600  | 0.4690          | 0.8697   | 0.8700   |
| 0.0825        | 106.52 | 9800  | 0.4758          | 0.8730   | 0.8734   |
| 0.0844        | 108.7  | 10000 | 0.4728          | 0.8703   | 0.8706   |


### Framework versions

- PEFT 0.9.0
- Transformers 4.38.2
- Pytorch 2.2.0+cu121
- Datasets 2.17.1
- Tokenizers 0.15.2