metadata

tags:
  - optimum
datasets:
  - banking77
metrics:
  - accuracy
model-index:
  - name: quantized-distilbert-banking77
    results:
      - task:
          name: Text Classification
          type: text-classification
        dataset:
          name: banking77
          type: banking77
        metrics:
          - name: Accuracy
            type: accuracy
            value: 0.9224

Quantized-distilbert-banking77

This model is a statically quantized version of optimum/distilbert-base-uncased-finetuned-banking77 on the banking77 dataset.

The model was created using the optimum-static-quantization notebook.

It achieves the following results on the evaluation set:

Accuracy

Vanilla model: 92.5%
Quantized model: 92.24%

The quantized model achieves 99.72% accuracy of the fp32 model

Latency

Payload sequence length: 128
Instance type: AWS c6i.xlarge

latency	vanilla transformers	quantized optimum model	improvement
p95	75.69ms	26.75ms	2.83x
avg	57.52ms	24.86ms	2.31x

How to use

from optimum.onnxruntime import ORTModelForSequenceClassification
from transformers import pipeline, AutoTokenizer

model = ORTModelForSequenceClassification.from_pretrained("philschmid/quantized-distilbert-banking77")
tokenizer = AutoTokenizer.from_pretrained("philschmid/quantized-distilbert-banking77")

remote_clx = pipeline("text-classification",model=model, tokenizer=tokenizer)

remote_clx("What is the exchange rate like on this app?")