Hugging Face's logo

mrm8488
/
deberta-v3-large-finetuned-mnli

Text Classification Transformers PyTorch
English deberta-v2 generated_from_trainer Eval Results Inference Endpoints
Model card Files Community
5
deberta-v3-large-finetuned-mnli / README.md
mrm8488's picture
mrm8488
Add verifyToken field to verify evaluation results are produced by Hugging Face's automatic model evaluator (#2)
51c6f1f
preview code
|
raw history blame contribute delete
No virus
7.13 kB
metadata
language:
  - en
license: mit
tags:
  - generated_from_trainer
datasets:
  - glue
metrics:
  - accuracy
widget:
  - text: She was badly wounded already. Another spear would take her down.
model-index:
  - name: deberta-v3-large-mnli-2
    results:
      - task:
          type: text-classification
          name: Text Classification
        dataset:
          name: GLUE MNLI
          type: glue
          args: mnli
        metrics:
          - type: accuracy
            value: 0.8949349064279902
            name: Accuracy
      - task:
          type: natural-language-inference
          name: Natural Language Inference
        dataset:
          name: glue
          type: glue
          config: mnli
          split: validation_matched
        metrics:
          - type: accuracy
            value: 0.9000509424350484
            name: Accuracy
            verified: true
            verifyToken: >-
              eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiYmU1NTE1YmYwOTA4NmQ3ZWE1MmM0ZDFiNDQ5YWIyMDMyZDhjZWMxYTQ3NGIxOWVkMTQxYTA3MTE2ZTUyYjg0ZiIsInZlcnNpb24iOjF9.UygjleiO4h0rlNa8KJIzJMy2VbMkLF-kB-YowCa_EhLKJQqRr9id5db81MyR_VV3ROrSdHVbCGIM9qxkPRbABg
          - type: precision
            value: 0.9000452542826349
            name: Precision Macro
            verified: true
            verifyToken: >-
              eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiN2EyMWYxY2ZlNTFhYWRhNjA4MzYxOTI4NDAzMjQwMmI4MTJmMWE3ZWEzZTQwMmMyZTM1MzIxYWEyYzVhNDlmMCIsInZlcnNpb24iOjF9.iq2CgF4ik1_DjPlbmFgxvscryy1NNQjTatCJhDu95sXMdlWkekPS6on3NyEaSDwptKyuTQiF4wh8WZDrfhO_Dw
          - type: precision
            value: 0.9000509424350484
            name: Precision Micro
            verified: true
            verifyToken: >-
              eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiYmY5NmE1MjU1Yzg3Mzk3MDJiNGUyMzM5NmYxYjljZjY1OTQ3NWE0MWM2MTZhYjQ4ZWFmY2FkODc4OThkMzIxMCIsInZlcnNpb24iOjF9.yN_8lq_IjeLU1WJknAkoj75MQajxLvsIsf_pOPFT0_Q77Vfhu0AsIdy1WDJcsAw08ziJoNpN_2LGDMBYJmwzCQ
          - type: precision
            value: 0.9014585350976404
            name: Precision Weighted
            verified: true
            verifyToken: >-
              eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiZTBkYWM4YTE3N2Q5ZmY5ZTRiMGQ1MDc5ODk2NjQwZDc0ODNkMjk3MjdjMjRlZDU2Yzk1MTliMzhmNjYzYzY2ZCIsInZlcnNpb24iOjF9.f9_fAM_a9LwSBwFgwaO5rdAYzV3wkhHq6yquugL1djRlbISZdpzZFWfJHcS-fvgMayYsklBK_ezbu0f7u7tyDg
          - type: recall
            value: 0.900253092056111
            name: Recall Macro
            verified: true
            verifyToken: >-
              eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiOTkwZTRmYzhjNDMyMDllNzFiYTNkMDdjN2E2NmEzOTdjMzAxNjdmMzg3OTFmN2IwZTlmYWY5MWQyMDUyNWRlMSIsInZlcnNpb24iOjF9.aWtX33vOHaGpePRZwO9dfTfWoWyXYCVAf8W1AlHXZto6Ve2HX9RLISTsALRMfNzX-7B6LYLh6qzusjf2xQ20Bw
          - type: recall
            value: 0.9000509424350484
            name: Recall Micro
            verified: true
            verifyToken: >-
              eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiMGFhYzVlZjQ3M2YyYjY1NTBiMGI4NmI4MTgwY2QzY2I3YmMyNjc3YmFhMDU1ZjNlY2FkMjQxOTg3YWYyYTU3ZiIsInZlcnNpb24iOjF9.wPD0-SL1vdG3_bi7cKh_hgVwVr1yV6zRYBzpGe6bDEzV5BYb5lCQoAebS5U1o2H4E4qi7zr2YNFEToNCRTqPBA
          - type: recall
            value: 0.9000509424350484
            name: Recall Weighted
            verified: true
            verifyToken: >-
              eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiNThmNjQ4MDY2ZTM3NjQyODQzMTZkNjgyMGNkNDE5MDMwOWJmMzhjZmZjNzllYjA4NmJiZDU3MzU3ODE0YjFhMyIsInZlcnNpb24iOjF9.yN9hb5VWX5ICIXdPBc0OD0BFHnzWv8rmmD--OEh6h1agGiRiyCdROo4saN5CQKiVlPBsHPliaoXra45Xi4gVAg
          - type: f1
            value: 0.8997940135019421
            name: F1 Macro
            verified: true
            verifyToken: >-
              eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiYmQzMWZhZTg1ODBmNWFiMGJiZDE5ODA2ZTA3NmUwZDcxMTQ1NzZjNDFiZDZkN2RmMmQ3YzRiMmI2Y2Q3MWRlNiIsInZlcnNpb24iOjF9.lr6jUSxXu6zKs_x-UQT7dL9_PzKTf50KUu7spTzRI6_SkaUyl9Ez0gR-O8bfzypaqkdxvtf7dsNFskpUvJ8wDQ
          - type: f1
            value: 0.9000509424350484
            name: F1 Micro
            verified: true
            verifyToken: >-
              eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiMWFiZjAzYjQ4NjFjMThjM2RlOGU1YzRjMmQzZTNhMDVjYWE3Njg5Y2QwMzc4YzY0ODNjOWUwMDJiNGU4ODk2MyIsInZlcnNpb24iOjF9.BsWoM2Mb4Kx5Lzm7b9GstHNuxGX7emrFNRcepgYNhjkeEhj3yJbvbboOaJuWMc9TdJEPr3o1PuNiu7zQ_vy_DQ
          - type: f1
            value: 0.9003949466748086
            name: F1 Weighted
            verified: true
            verifyToken: >-
              eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiNWQ1NjA2Njc0Njk2YzY0MzIwYTYwMWM5MTZhNzhhZDY2ODgyYzVlODlmN2Q2MjRjNzhhNzMyZDQ1ZmYwMjdlMyIsInZlcnNpb24iOjF9.Xdl4G3GaOXzCRhaoDf_sJThoEQLmlGyf4efJCYFKXCe1DfNb4qOl-_h9LuE3-iacvusjIJFIquhQ7YsLtqbrCg
          - type: loss
            value: 0.6493226289749146
            name: loss
            verified: true
            verifyToken: >-
              eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiZWU0ZGM5MWE2Mjk3NDI5ZGNkZmFhM2IxYmFiZjVkMjdiNTE4NzA5YWMxNDcxOWYxYjA2MmQ3ZmE1Yzk5M2E2OCIsInZlcnNpb24iOjF9.gsO8l1_9H89OaztnG6rhNuOY-ssmafoUSwuyNRPR5TjqwrimWk4S6k2uCSSoV9h_JvtliFQ94aZhgSB2lGxWCg

DeBERTa-v3-large fine-tuned on MNLI

This model is a fine-tuned version of microsoft/deberta-v3-large on the GLUE MNLI dataset. It achieves the following results on the evaluation set:

  • Loss: 0.6763
  • Accuracy: 0.8949

Model description

DeBERTa improves the BERT and RoBERTa models using disentangled attention and enhanced mask decoder. With those two improvements, DeBERTa out perform RoBERTa on a majority of NLU tasks with 80GB training data.

In DeBERTa V3, we further improved the efficiency of DeBERTa using ELECTRA-Style pre-training with Gradient Disentangled Embedding Sharing. Compared to DeBERTa, our V3 version significantly improves the model performance on downstream tasks. You can find more technique details about the new model from our paper.

Please check the official repository for more implementation details and updates.

The DeBERTa V3 large model comes with 24 layers and a hidden size of 1024. It has 304M backbone parameters with a vocabulary containing 128K tokens which introduces 131M parameters in the Embedding layer. This model was trained using the 160GB data as DeBERTa V2.

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

  • learning_rate: 3e-05
  • train_batch_size: 16
  • eval_batch_size: 8
  • seed: 42
  • optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
  • lr_scheduler_type: linear
  • num_epochs: 5.0
  • mixed_precision_training: Native AMP

Training results

Training Loss Epoch Step Validation Loss Accuracy
0.3676 1.0 24544 0.3761 0.8681
0.2782 2.0 49088 0.3605 0.8881
0.1986 3.0 73632 0.4672 0.8894
0.1299 4.0 98176 0.5248 0.8967
0.0643 5.0 122720 0.6489 0.8999

Framework versions

  • Transformers 4.13.0.dev0
  • Pytorch 1.10.0+cu111
  • Datasets 1.16.1
  • Tokenizers 0.10.3