---
tags:
- generated_from_trainer
datasets:
- vblagoje/cc_news
model-index:
- name: pretrained-bert
  results: []
---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

# pretrained-bert

This model is a fine-tuned version of [](https://huggingface.co/) on the cc_news dataset.
It achieves the following results on the evaluation set:
- Loss: 5.7000

## Model description

More information needed

## Intended uses & limitations

More information needed

## Training and evaluation data

More information needed

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 5e-05
- train_batch_size: 4
- eval_batch_size: 4
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 5

### Training results

| Training Loss | Epoch | Step  | Validation Loss |
|:-------------:|:-----:|:-----:|:---------------:|
| 7.1956        | 0.08  | 1000  | 7.0109          |
| 6.9483        | 0.17  | 2000  | 6.8538          |
| 6.8304        | 0.25  | 3000  | 6.7408          |
| 6.7429        | 0.33  | 4000  | 6.6812          |
| 6.6812        | 0.42  | 5000  | 6.6208          |
| 6.6102        | 0.5   | 6000  | 6.5837          |
| 6.5974        | 0.58  | 7000  | 6.5340          |
| 6.54          | 0.67  | 8000  | 6.5080          |
| 6.5184        | 0.75  | 9000  | 6.4795          |
| 6.4937        | 0.83  | 10000 | 6.4346          |
| 6.4671        | 0.92  | 11000 | 6.4364          |
| 6.4347        | 1.0   | 12000 | 6.4180          |
| 6.3966        | 1.08  | 13000 | 6.4009          |
| 6.3911        | 1.17  | 14000 | 6.3625          |
| 6.3594        | 1.25  | 15000 | 6.3390          |
| 6.3437        | 1.33  | 16000 | 6.3383          |
| 6.2964        | 1.42  | 17000 | 6.2872          |
| 6.3101        | 1.5   | 18000 | 6.2789          |
| 6.3023        | 1.58  | 19000 | 6.2913          |
| 6.2916        | 1.67  | 20000 | 6.2443          |
| 6.2647        | 1.75  | 21000 | 6.2324          |
| 6.2418        | 1.83  | 22000 | 6.2291          |
| 6.2432        | 1.92  | 23000 | 6.2088          |
| 6.2269        | 2.0   | 24000 | 6.1954          |
| 6.1794        | 2.08  | 25000 | 6.1979          |
| 6.1813        | 2.17  | 26000 | 6.1586          |
| 6.186         | 2.25  | 27000 | 6.1581          |
| 6.1497        | 2.33  | 28000 | 6.1378          |
| 6.1253        | 2.42  | 29000 | 6.1364          |
| 6.1278        | 2.5   | 30000 | 6.1096          |
| 6.0937        | 2.58  | 31000 | 6.0842          |
| 6.0982        | 2.67  | 32000 | 6.0802          |
| 6.0834        | 2.75  | 33000 | 6.0576          |
| 6.0595        | 2.83  | 34000 | 6.0290          |
| 6.0427        | 2.92  | 35000 | 6.0079          |
| 6.0553        | 3.0   | 36000 | 6.0322          |
| 5.9979        | 3.08  | 37000 | 5.9859          |
| 5.9882        | 3.17  | 38000 | 5.9754          |
| 5.9933        | 3.25  | 39000 | 5.9619          |
| 5.9599        | 3.33  | 40000 | 5.9738          |
| 5.9597        | 3.42  | 41000 | 5.9464          |
| 5.95          | 3.5   | 42000 | 5.9281          |
| 5.9613        | 3.58  | 43000 | 5.9190          |
| 5.9362        | 3.67  | 44000 | 5.9000          |
| 5.9186        | 3.75  | 45000 | 5.8801          |
| 5.9263        | 3.83  | 46000 | 5.8398          |
| 5.8747        | 3.92  | 47000 | 5.8371          |
| 5.8566        | 4.0   | 48000 | 5.8147          |
| 5.8434        | 4.08  | 49000 | 5.8102          |
| 5.8194        | 4.17  | 50000 | 5.7867          |
| 5.8467        | 4.25  | 51000 | 5.7718          |
| 5.8149        | 4.33  | 52000 | 5.7629          |
| 5.7929        | 4.42  | 53000 | 5.7506          |
| 5.7763        | 4.5   | 54000 | 5.7356          |
| 5.7757        | 4.58  | 55000 | 5.7236          |
| 5.7624        | 4.67  | 56000 | 5.6957          |
| 5.7464        | 4.75  | 57000 | 5.7163          |
| 5.778         | 4.83  | 58000 | 5.7008          |
| 5.7322        | 4.92  | 59000 | 5.6870          |
| 5.7392        | 5.0   | 60000 | 5.7000          |


### Framework versions

- Transformers 4.30.2
- Pytorch 2.0.0
- Datasets 2.1.0
- Tokenizers 0.13.3