metadata
license: mit
language:
- en
tags:
- sparse sparsity quantized onnx embeddings int8
model-index:
- name: bge-base-en-v1.5-sparse
results:
- task:
type: Classification
dataset:
type: mteb/amazon_counterfactual
name: MTEB AmazonCounterfactualClassification (en)
config: en
split: test
revision: e8379541af4e31359cca9fbcf4b00f2671dba205
metrics:
- type: accuracy
value: 75.38805970149254
- type: ap
value: 38.80643435437097
- type: f1
value: 69.52906891019036
- task:
type: Classification
dataset:
type: mteb/amazon_polarity
name: MTEB AmazonPolarityClassification
config: default
split: test
revision: e2d317d38cd51312af73b3d32a06d1a08b442046
metrics:
- type: accuracy
value: 90.72759999999998
- type: ap
value: 87.07910150764239
- type: f1
value: 90.71025910882096
- task:
type: Classification
dataset:
type: mteb/amazon_reviews_multi
name: MTEB AmazonReviewsClassification (en)
config: en
split: test
revision: 1399c76144fd37290681b995c656ef9b2e06e26d
metrics:
- type: accuracy
value: 45.494
- type: f1
value: 44.917953161904805
- task:
type: Classification
dataset:
type: mteb/banking77
name: MTEB Banking77Classification
config: default
split: test
revision: 0fd18e25b25c072e09e0d92ab615fda904d66300
metrics:
- type: accuracy
value: 84.29545454545455
- type: f1
value: 84.26780483160312
- task:
type: Classification
dataset:
type: mteb/emotion
name: MTEB EmotionClassification
config: default
split: test
revision: 4f58c6b202a23cf9a4da393831edf4f9183cad37
metrics:
- type: accuracy
value: 46.705
- type: f1
value: 41.82618717355017
- task:
type: Classification
dataset:
type: mteb/imdb
name: MTEB ImdbClassification
config: default
split: test
revision: 3d86128a09e091d6018b6d26cad27f2739fc2db7
metrics:
- type: accuracy
value: 83.14760000000001
- type: ap
value: 77.40813245635195
- type: f1
value: 83.08648833100911
- task:
type: Classification
dataset:
type: mteb/mtop_domain
name: MTEB MTOPDomainClassification (en)
config: en
split: test
revision: d80d48c1eb48d3562165c59d59d0034df9fff0bf
metrics:
- type: accuracy
value: 92.0519835841313
- type: f1
value: 91.73392170858916
- task:
type: Classification
dataset:
type: mteb/mtop_intent
name: MTEB MTOPIntentClassification (en)
config: en
split: test
revision: ae001d0e6b1228650b7bd1c2c65fb50ad11a8aba
metrics:
- type: accuracy
value: 72.48974008207935
- type: f1
value: 54.812872972777505
- task:
type: Classification
dataset:
type: mteb/amazon_massive_intent
name: MTEB MassiveIntentClassification (en)
config: en
split: test
revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7
metrics:
- type: accuracy
value: 73.17753866846
- type: f1
value: 71.51091282373878
- task:
type: Classification
dataset:
type: mteb/amazon_massive_scenario
name: MTEB MassiveScenarioClassification (en)
config: en
split: test
revision: 7d571f92784cd94a019292a1f45445077d0ef634
metrics:
- type: accuracy
value: 77.5353059852051
- type: f1
value: 77.42427561340143
- task:
type: Classification
dataset:
type: mteb/toxic_conversations_50k
name: MTEB ToxicConversationsClassification
config: default
split: test
revision: d7c0de2777da35d6aae2200a62c6e0e5af397c4c
metrics:
- type: accuracy
value: 70.917
- type: ap
value: 13.760770628090576
- type: f1
value: 54.23887489664618
- task:
type: Classification
dataset:
type: mteb/tweet_sentiment_extraction
name: MTEB TweetSentimentExtractionClassification
config: default
split: test
revision: d604517c81ca91fe16a244d1248fc021f9ecee7a
metrics:
- type: accuracy
value: 59.49349179400113
- type: f1
value: 59.815392064510775
This is the sparsified ONNX variant of the bge-base-en-v1.5 embeddings model created with DeepSparse Optimum for ONNX export/inference pipeline and Neural Magic's Sparsify for one-shot quantization (INT8) and unstructured pruning (50%).
Current list of sparse and quantized bge ONNX models:
| Links | Sparsification Method |
|---|---|
| zeroshot/bge-large-en-v1.5-sparse | Quantization (INT8) & 50% Pruning |
| zeroshot/bge-large-en-v1.5-quant | Quantization (INT8) |
| zeroshot/bge-base-en-v1.5-sparse | Quantization (INT8) & 50% Pruning |
| zeroshot/bge-base-en-v1.5-quant | Quantization (INT8) |
| zeroshot/bge-small-en-v1.5-sparse | Quantization (INT8) & 50% Pruning |
| zeroshot/bge-small-en-v1.5-quant | Quantization (INT8) |
For general questions on these models and sparsification methods, reach out to the engineering team on our community Slack.
