Hugging Face's logo

postbot
/
bert_uncased_L-2_H-768_A-12-mlm-multi-emails-hq

Fill-Mask
Transformers
PyTorch
Safetensors
English
bert
Generated from Trainer
Inference Endpoints
Model card Files Community
1
bert_uncased_L-2_H-768_A-12-mlm-multi-emails-hq / README.md
pszemraj's picture
pszemraj
Update README.md
61302e6
preview code
|
raw
history blame contribute delete
2.51 kB
metadata
license: apache-2.0
tags:
  - generated_from_trainer
metrics:
  - accuracy
model-index:
  - name: bert_uncased_L-2_H-768_A-12-mlm-multi-emails-hq
    results: []
datasets:
  - postbot/multi-emails-hq
language:
  - en
pipeline_tag: fill-mask
widget:
  - text: Can you please send me the [MASK] by the end of the day?
    example_title: end of day
  - text: >-
      I hope this email finds you well. I wanted to follow up on our [MASK]
      yesterday.
    example_title: follow-up
  - text: The meeting has been rescheduled to [MASK].
    example_title: reschedule
  - text: Please let me know if you need any further [MASK] regarding the project.
    example_title: further help
  - text: >-
      I appreciate your prompt response to my previous email. Can you provide an
      update on the [MASK] by tomorrow?
    example_title: provide update
  - text: Paris is the [MASK] of France.
    example_title: paris (default)
  - text: The goal of life is [MASK].
    example_title: goal of life (default)

bert_uncased_L-2_H-768_A-12-mlm-multi-emails-hq

This model is a fine-tuned version of google/bert_uncased_L-2_H-768_A-12 on the None dataset. It achieves the following results on the evaluation set:

  • Loss: 1.9133
  • Accuracy: 0.6452

Model description

Small BERT, uncased. 155 MB.

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

  • learning_rate: 0.0003
  • train_batch_size: 8
  • eval_batch_size: 2
  • seed: 42
  • distributed_type: multi-GPU
  • gradient_accumulation_steps: 16
  • total_train_batch_size: 128
  • optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
  • lr_scheduler_type: cosine
  • lr_scheduler_warmup_ratio: 0.05
  • num_epochs: 5.0
  • mixed_precision_training: Native AMP

Training results

Training Loss Epoch Step Validation Loss Accuracy
2.3053 0.99 141 2.1758 0.6064
2.1556 1.99 282 2.0587 0.6237
2.0616 2.99 423 1.9780 0.6355
2.0084 3.99 564 1.9317 0.6422
1.9621 4.99 705 1.9133 0.6452

Framework versions

  • Transformers 4.27.0.dev0
  • Pytorch 2.0.0.dev20230129+cu118
  • Datasets 2.8.0
  • Tokenizers 0.13.1