cartesinus's picture
Update README.md
52f2f02
metadata
language:
  - en
license: mit
tags:
  - generated_from_trainer
  - nlu
  - intent-classification
datasets:
  - AmazonScience/massive
metrics:
  - accuracy
  - f1
base_model: xlm-roberta-base
model-index:
  - name: xlm-r-base-amazon-massive-intent
    results:
      - task:
          type: intent-classification
          name: intent-classification
        dataset:
          name: MASSIVE
          type: AmazonScience/massive
          split: test
        metrics:
          - type: f1
            value: 0.8775
            name: F1

xlm-r-base-amazon-massive-intent

This model is a fine-tuned version of xlm-roberta-base on Amazon Massive dataset (only en-US subset). It achieves the following results on the evaluation set:

  • Loss: 0.5439
  • Accuracy: 0.8775
  • F1: 0.8775

Results

domain train-accuracy test-accuracy
alarm 0.967 0.9846
audio 0.7458 0.659
calendar 0.9797 0.3181
cooking 0.9714 0.9571
datetime 0.9777 0.9402
email 0.9727 0.9296
general 0.8952 0.5949
iot 0.9329 0.9122
list 0.9792 0.9538
music 0.9355 0.8837
news 0.9607 0.8764
play 0.9419 0.874
qa 0.9677 0.8591
recommendation 0.9515 0.8764
social 0.9671 0.8932
takeaway 0.9192 0.8478
transport 0.9425 0.9193
weather 0.9895 0.93

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

  • learning_rate: 2e-05
  • train_batch_size: 16
  • eval_batch_size: 16
  • seed: 42
  • optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
  • lr_scheduler_type: linear
  • num_epochs: 5

Training results

Training Loss Epoch Step Validation Loss Accuracy F1
2.734 1.0 720 1.1883 0.7196 0.7196
1.2774 2.0 1440 0.7162 0.8342 0.8342
0.6301 3.0 2160 0.5817 0.8672 0.8672
0.4901 4.0 2880 0.5555 0.8770 0.8770
0.3398 5.0 3600 0.5439 0.8775 0.8775

Framework versions

  • Transformers 4.22.1
  • Pytorch 1.12.1+cu113
  • Datasets 2.5.1
  • Tokenizers 0.12.1

Citation

@article{kubis2023back,
  title={Back Transcription as a Method for Evaluating Robustness of Natural Language Understanding Models to Speech Recognition Errors},
  author={Kubis, Marek and Sk{\'o}rzewski, Pawe{\l} and Sowa{\'n}ski, Marcin and Zi{\k{e}}tkiewicz, Tomasz},
  journal={arXiv preprint arXiv:2310.16609},
  year={2023}
  eprint={2310.16609},
  archivePrefix={arXiv},
}