File size: 5,227 Bytes
19b0570 eae4e05 19b0570 eae4e05 19b0570 eae4e05 19b0570 eae4e05 19b0570 eae4e05 19b0570 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 |
---
license: mit
tags:
- generated_from_trainer
metrics:
- f1
model-index:
- name: minilm-finetuned-movie
results: []
---
<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->
# minilm-finetuned-movie
This model is a fine-tuned version of [microsoft/miniLM-L12-H384-uncased](https://huggingface.co/microsoft/miniLM-L12-H384-uncased) on sasingh192/movie-review dataset.
It achieves the following results on the evaluation set:
- Loss: 0.0451
- F1: 0.9856
## Model description
This model can be used to categorize a movie review into of the following categories:
0 - negative
1 - somewhat negative
2 - neutral
3 - somewhat positive
4 - positive
## Intended uses & limitations
The fined model is based on the finetuning of the model devloped by Wang et al.
@misc{wang2020minilm,
title={MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers},
author={Wenhui Wang and Furu Wei and Li Dong and Hangbo Bao and Nan Yang and Ming Zhou},
year={2020},
eprint={2002.10957},
archivePrefix={arXiv},
primaryClass={cs.CL}
}
## Training and evaluation data
sasingh192/movie-review dataset contains a column 'TrainValTest'. The values provied in this columns are 'Train', 'Val', and 'Test'.
The dataset can be filtered for the 'Train' values to train the model. Evaluation can be perfored on the data filtered by 'Val'. 'Test' is used as a blind test for kaggle.
## Training procedure
Training details are listed below.
### Training hyperparameters
The following hyperparameters were used during training:
- learning_rate: 2e-05
- train_batch_size: 64
- eval_batch_size: 64
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 50
- mixed_precision_training: Native AMP
### Training results
| Training Loss | Epoch | Step | Validation Loss | F1 |
|:-------------:|:-----:|:-----:|:---------------:|:------:|
| 0.9623 | 1.0 | 1946 | 0.7742 | 0.6985 |
| 0.7969 | 2.0 | 3892 | 0.7289 | 0.7094 |
| 0.74 | 3.0 | 5838 | 0.6479 | 0.7476 |
| 0.7012 | 4.0 | 7784 | 0.6263 | 0.7550 |
| 0.6689 | 5.0 | 9730 | 0.5823 | 0.7762 |
| 0.6416 | 6.0 | 11676 | 0.5796 | 0.7673 |
| 0.6149 | 7.0 | 13622 | 0.5324 | 0.7912 |
| 0.5939 | 8.0 | 15568 | 0.5189 | 0.7986 |
| 0.5714 | 9.0 | 17514 | 0.4793 | 0.8184 |
| 0.5495 | 10.0 | 19460 | 0.4566 | 0.8249 |
| 0.5297 | 11.0 | 21406 | 0.4155 | 0.8475 |
| 0.5101 | 12.0 | 23352 | 0.4063 | 0.8494 |
| 0.4924 | 13.0 | 25298 | 0.3829 | 0.8571 |
| 0.4719 | 14.0 | 27244 | 0.4032 | 0.8449 |
| 0.4552 | 15.0 | 29190 | 0.3447 | 0.8720 |
| 0.4382 | 16.0 | 31136 | 0.3581 | 0.8610 |
| 0.421 | 17.0 | 33082 | 0.3095 | 0.8835 |
| 0.4038 | 18.0 | 35028 | 0.2764 | 0.9002 |
| 0.3883 | 19.0 | 36974 | 0.2610 | 0.9051 |
| 0.3745 | 20.0 | 38920 | 0.2533 | 0.9064 |
| 0.3616 | 21.0 | 40866 | 0.2601 | 0.9005 |
| 0.345 | 22.0 | 42812 | 0.2085 | 0.9267 |
| 0.3314 | 23.0 | 44758 | 0.2421 | 0.9069 |
| 0.3178 | 24.0 | 46704 | 0.2006 | 0.9268 |
| 0.3085 | 25.0 | 48650 | 0.1846 | 0.9326 |
| 0.2964 | 26.0 | 50596 | 0.1492 | 0.9490 |
| 0.2855 | 27.0 | 52542 | 0.1664 | 0.9376 |
| 0.2737 | 28.0 | 54488 | 0.1309 | 0.9560 |
| 0.2641 | 29.0 | 56434 | 0.1318 | 0.9562 |
| 0.2541 | 30.0 | 58380 | 0.1490 | 0.9440 |
| 0.2462 | 31.0 | 60326 | 0.1195 | 0.9575 |
| 0.234 | 32.0 | 62272 | 0.1054 | 0.9640 |
| 0.2273 | 33.0 | 64218 | 0.1054 | 0.9631 |
| 0.2184 | 34.0 | 66164 | 0.0971 | 0.9662 |
| 0.214 | 35.0 | 68110 | 0.0902 | 0.9689 |
| 0.2026 | 36.0 | 70056 | 0.0846 | 0.9699 |
| 0.1973 | 37.0 | 72002 | 0.0819 | 0.9705 |
| 0.1934 | 38.0 | 73948 | 0.0810 | 0.9716 |
| 0.1884 | 39.0 | 75894 | 0.0724 | 0.9746 |
| 0.1796 | 40.0 | 77840 | 0.0737 | 0.9743 |
| 0.1779 | 41.0 | 79786 | 0.0665 | 0.9773 |
| 0.1703 | 42.0 | 81732 | 0.0568 | 0.9811 |
| 0.1638 | 43.0 | 83678 | 0.0513 | 0.9843 |
| 0.1601 | 44.0 | 85624 | 0.0575 | 0.9802 |
| 0.1593 | 45.0 | 87570 | 0.0513 | 0.9835 |
| 0.1559 | 46.0 | 89516 | 0.0474 | 0.9851 |
| 0.1514 | 47.0 | 91462 | 0.0477 | 0.9847 |
| 0.1473 | 48.0 | 93408 | 0.0444 | 0.9858 |
| 0.1462 | 49.0 | 95354 | 0.0449 | 0.9855 |
| 0.1458 | 50.0 | 97300 | 0.0451 | 0.9856 |
### Framework versions
- Transformers 4.29.2
- Pytorch 2.0.1
- Datasets 2.12.0
- Tokenizers 0.13.2
|