|
--- |
|
tags: |
|
- feature-extraction |
|
- mteb |
|
pipeline_tag: feature-extraction |
|
model-index: |
|
- name: dragon-plus |
|
results: |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: arguana |
|
name: MTEB ArguAna |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 22.973 |
|
- type: map_at_10 |
|
value: 38.242 |
|
- type: map_at_100 |
|
value: 39.326 |
|
- type: map_at_1000 |
|
value: 39.342 |
|
- type: map_at_3 |
|
value: 33.144 |
|
- type: map_at_5 |
|
value: 35.818 |
|
- type: mrr_at_1 |
|
value: 23.115 |
|
- type: mrr_at_10 |
|
value: 38.31 |
|
- type: mrr_at_100 |
|
value: 39.387 |
|
- type: mrr_at_1000 |
|
value: 39.403 |
|
- type: mrr_at_3 |
|
value: 33.167 |
|
- type: mrr_at_5 |
|
value: 35.856 |
|
- type: ndcg_at_1 |
|
value: 22.973 |
|
- type: ndcg_at_10 |
|
value: 47.251 |
|
- type: ndcg_at_100 |
|
value: 51.937 |
|
- type: ndcg_at_1000 |
|
value: 52.288000000000004 |
|
- type: ndcg_at_3 |
|
value: 36.569 |
|
- type: ndcg_at_5 |
|
value: 41.396 |
|
- type: precision_at_1 |
|
value: 22.973 |
|
- type: precision_at_10 |
|
value: 7.632 |
|
- type: precision_at_100 |
|
value: 0.9690000000000001 |
|
- type: precision_at_1000 |
|
value: 0.1 |
|
- type: precision_at_3 |
|
value: 15.504999999999999 |
|
- type: precision_at_5 |
|
value: 11.65 |
|
- type: recall_at_1 |
|
value: 22.973 |
|
- type: recall_at_10 |
|
value: 76.31599999999999 |
|
- type: recall_at_100 |
|
value: 96.942 |
|
- type: recall_at_1000 |
|
value: 99.57300000000001 |
|
- type: recall_at_3 |
|
value: 46.515 |
|
- type: recall_at_5 |
|
value: 58.25 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: BeIR/cqadupstack |
|
name: MTEB CQADupstackAndroidRetrieval |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 28.793000000000003 |
|
- type: map_at_10 |
|
value: 38.686 |
|
- type: map_at_100 |
|
value: 39.848 |
|
- type: map_at_1000 |
|
value: 39.989999999999995 |
|
- type: map_at_3 |
|
value: 35.437000000000005 |
|
- type: map_at_5 |
|
value: 37.067 |
|
- type: mrr_at_1 |
|
value: 35.05 |
|
- type: mrr_at_10 |
|
value: 43.903999999999996 |
|
- type: mrr_at_100 |
|
value: 44.612 |
|
- type: mrr_at_1000 |
|
value: 44.669 |
|
- type: mrr_at_3 |
|
value: 41.321000000000005 |
|
- type: mrr_at_5 |
|
value: 42.573 |
|
- type: ndcg_at_1 |
|
value: 35.05 |
|
- type: ndcg_at_10 |
|
value: 44.564 |
|
- type: ndcg_at_100 |
|
value: 49.252 |
|
- type: ndcg_at_1000 |
|
value: 51.791 |
|
- type: ndcg_at_3 |
|
value: 39.576 |
|
- type: ndcg_at_5 |
|
value: 41.426 |
|
- type: precision_at_1 |
|
value: 35.05 |
|
- type: precision_at_10 |
|
value: 8.455 |
|
- type: precision_at_100 |
|
value: 1.3299999999999998 |
|
- type: precision_at_1000 |
|
value: 0.187 |
|
- type: precision_at_3 |
|
value: 18.645999999999997 |
|
- type: precision_at_5 |
|
value: 13.247 |
|
- type: recall_at_1 |
|
value: 28.793000000000003 |
|
- type: recall_at_10 |
|
value: 56.351 |
|
- type: recall_at_100 |
|
value: 76.542 |
|
- type: recall_at_1000 |
|
value: 93.14099999999999 |
|
- type: recall_at_3 |
|
value: 41.581 |
|
- type: recall_at_5 |
|
value: 47.066 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: BeIR/cqadupstack |
|
name: MTEB CQADupstackEnglishRetrieval |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 29.828 |
|
- type: map_at_10 |
|
value: 39.312999999999995 |
|
- type: map_at_100 |
|
value: 40.487 |
|
- type: map_at_1000 |
|
value: 40.607 |
|
- type: map_at_3 |
|
value: 36.525 |
|
- type: map_at_5 |
|
value: 38.121 |
|
- type: mrr_at_1 |
|
value: 37.197 |
|
- type: mrr_at_10 |
|
value: 45.091 |
|
- type: mrr_at_100 |
|
value: 45.726 |
|
- type: mrr_at_1000 |
|
value: 45.769999999999996 |
|
- type: mrr_at_3 |
|
value: 42.856 |
|
- type: mrr_at_5 |
|
value: 44.056 |
|
- type: ndcg_at_1 |
|
value: 37.197 |
|
- type: ndcg_at_10 |
|
value: 44.737 |
|
- type: ndcg_at_100 |
|
value: 49.02 |
|
- type: ndcg_at_1000 |
|
value: 51.052 |
|
- type: ndcg_at_3 |
|
value: 40.685 |
|
- type: ndcg_at_5 |
|
value: 42.519 |
|
- type: precision_at_1 |
|
value: 37.197 |
|
- type: precision_at_10 |
|
value: 8.363 |
|
- type: precision_at_100 |
|
value: 1.329 |
|
- type: precision_at_1000 |
|
value: 0.179 |
|
- type: precision_at_3 |
|
value: 19.533 |
|
- type: precision_at_5 |
|
value: 13.732 |
|
- type: recall_at_1 |
|
value: 29.828 |
|
- type: recall_at_10 |
|
value: 54.339000000000006 |
|
- type: recall_at_100 |
|
value: 72.217 |
|
- type: recall_at_1000 |
|
value: 85.185 |
|
- type: recall_at_3 |
|
value: 42.331 |
|
- type: recall_at_5 |
|
value: 47.612 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: BeIR/cqadupstack |
|
name: MTEB CQADupstackGamingRetrieval |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 37.919000000000004 |
|
- type: map_at_10 |
|
value: 49.225 |
|
- type: map_at_100 |
|
value: 50.306 |
|
- type: map_at_1000 |
|
value: 50.364 |
|
- type: map_at_3 |
|
value: 46.459 |
|
- type: map_at_5 |
|
value: 48.173 |
|
- type: mrr_at_1 |
|
value: 43.072 |
|
- type: mrr_at_10 |
|
value: 52.437 |
|
- type: mrr_at_100 |
|
value: 53.2 |
|
- type: mrr_at_1000 |
|
value: 53.233 |
|
- type: mrr_at_3 |
|
value: 50.219 |
|
- type: mrr_at_5 |
|
value: 51.629999999999995 |
|
- type: ndcg_at_1 |
|
value: 43.072 |
|
- type: ndcg_at_10 |
|
value: 54.468 |
|
- type: ndcg_at_100 |
|
value: 58.912 |
|
- type: ndcg_at_1000 |
|
value: 60.179 |
|
- type: ndcg_at_3 |
|
value: 49.836999999999996 |
|
- type: ndcg_at_5 |
|
value: 52.371 |
|
- type: precision_at_1 |
|
value: 43.072 |
|
- type: precision_at_10 |
|
value: 8.52 |
|
- type: precision_at_100 |
|
value: 1.168 |
|
- type: precision_at_1000 |
|
value: 0.133 |
|
- type: precision_at_3 |
|
value: 21.923000000000002 |
|
- type: precision_at_5 |
|
value: 14.997 |
|
- type: recall_at_1 |
|
value: 37.919000000000004 |
|
- type: recall_at_10 |
|
value: 66.682 |
|
- type: recall_at_100 |
|
value: 85.81 |
|
- type: recall_at_1000 |
|
value: 94.812 |
|
- type: recall_at_3 |
|
value: 54.515 |
|
- type: recall_at_5 |
|
value: 60.684000000000005 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: BeIR/cqadupstack |
|
name: MTEB CQADupstackGisRetrieval |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 21.04 |
|
- type: map_at_10 |
|
value: 27.665 |
|
- type: map_at_100 |
|
value: 28.716 |
|
- type: map_at_1000 |
|
value: 28.794999999999998 |
|
- type: map_at_3 |
|
value: 25.338 |
|
- type: map_at_5 |
|
value: 26.815 |
|
- type: mrr_at_1 |
|
value: 22.712 |
|
- type: mrr_at_10 |
|
value: 29.447000000000003 |
|
- type: mrr_at_100 |
|
value: 30.457 |
|
- type: mrr_at_1000 |
|
value: 30.522 |
|
- type: mrr_at_3 |
|
value: 27.119 |
|
- type: mrr_at_5 |
|
value: 28.582 |
|
- type: ndcg_at_1 |
|
value: 22.712 |
|
- type: ndcg_at_10 |
|
value: 31.77 |
|
- type: ndcg_at_100 |
|
value: 37.104 |
|
- type: ndcg_at_1000 |
|
value: 39.371 |
|
- type: ndcg_at_3 |
|
value: 27.171 |
|
- type: ndcg_at_5 |
|
value: 29.698999999999998 |
|
- type: precision_at_1 |
|
value: 22.712 |
|
- type: precision_at_10 |
|
value: 4.859 |
|
- type: precision_at_100 |
|
value: 0.7929999999999999 |
|
- type: precision_at_1000 |
|
value: 0.10300000000000001 |
|
- type: precision_at_3 |
|
value: 11.299 |
|
- type: precision_at_5 |
|
value: 8.203000000000001 |
|
- type: recall_at_1 |
|
value: 21.04 |
|
- type: recall_at_10 |
|
value: 42.848000000000006 |
|
- type: recall_at_100 |
|
value: 67.694 |
|
- type: recall_at_1000 |
|
value: 85.179 |
|
- type: recall_at_3 |
|
value: 30.54 |
|
- type: recall_at_5 |
|
value: 36.555 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: BeIR/cqadupstack |
|
name: MTEB CQADupstackMathematicaRetrieval |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 13.403 |
|
- type: map_at_10 |
|
value: 19.663 |
|
- type: map_at_100 |
|
value: 20.799 |
|
- type: map_at_1000 |
|
value: 20.915 |
|
- type: map_at_3 |
|
value: 17.465 |
|
- type: map_at_5 |
|
value: 18.665000000000003 |
|
- type: mrr_at_1 |
|
value: 16.418 |
|
- type: mrr_at_10 |
|
value: 23.394000000000002 |
|
- type: mrr_at_100 |
|
value: 24.363 |
|
- type: mrr_at_1000 |
|
value: 24.44 |
|
- type: mrr_at_3 |
|
value: 20.916 |
|
- type: mrr_at_5 |
|
value: 22.241 |
|
- type: ndcg_at_1 |
|
value: 16.418 |
|
- type: ndcg_at_10 |
|
value: 24.013 |
|
- type: ndcg_at_100 |
|
value: 29.62 |
|
- type: ndcg_at_1000 |
|
value: 32.518 |
|
- type: ndcg_at_3 |
|
value: 19.747 |
|
- type: ndcg_at_5 |
|
value: 21.689 |
|
- type: precision_at_1 |
|
value: 16.418 |
|
- type: precision_at_10 |
|
value: 4.515000000000001 |
|
- type: precision_at_100 |
|
value: 0.8410000000000001 |
|
- type: precision_at_1000 |
|
value: 0.123 |
|
- type: precision_at_3 |
|
value: 9.411 |
|
- type: precision_at_5 |
|
value: 6.965000000000001 |
|
- type: recall_at_1 |
|
value: 13.403 |
|
- type: recall_at_10 |
|
value: 33.731 |
|
- type: recall_at_100 |
|
value: 58.743 |
|
- type: recall_at_1000 |
|
value: 79.343 |
|
- type: recall_at_3 |
|
value: 22.148 |
|
- type: recall_at_5 |
|
value: 26.998 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: BeIR/cqadupstack |
|
name: MTEB CQADupstackPhysicsRetrieval |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 25.782 |
|
- type: map_at_10 |
|
value: 34.891 |
|
- type: map_at_100 |
|
value: 36.186 |
|
- type: map_at_1000 |
|
value: 36.303999999999995 |
|
- type: map_at_3 |
|
value: 32.099 |
|
- type: map_at_5 |
|
value: 33.777 |
|
- type: mrr_at_1 |
|
value: 30.895 |
|
- type: mrr_at_10 |
|
value: 40.049 |
|
- type: mrr_at_100 |
|
value: 40.953 |
|
- type: mrr_at_1000 |
|
value: 41.0 |
|
- type: mrr_at_3 |
|
value: 37.424 |
|
- type: mrr_at_5 |
|
value: 39.07 |
|
- type: ndcg_at_1 |
|
value: 30.895 |
|
- type: ndcg_at_10 |
|
value: 40.436 |
|
- type: ndcg_at_100 |
|
value: 46.046 |
|
- type: ndcg_at_1000 |
|
value: 48.324 |
|
- type: ndcg_at_3 |
|
value: 35.66 |
|
- type: ndcg_at_5 |
|
value: 38.167 |
|
- type: precision_at_1 |
|
value: 30.895 |
|
- type: precision_at_10 |
|
value: 7.151000000000001 |
|
- type: precision_at_100 |
|
value: 1.171 |
|
- type: precision_at_1000 |
|
value: 0.155 |
|
- type: precision_at_3 |
|
value: 16.619 |
|
- type: precision_at_5 |
|
value: 11.935 |
|
- type: recall_at_1 |
|
value: 25.782 |
|
- type: recall_at_10 |
|
value: 52.013 |
|
- type: recall_at_100 |
|
value: 75.736 |
|
- type: recall_at_1000 |
|
value: 90.823 |
|
- type: recall_at_3 |
|
value: 38.763 |
|
- type: recall_at_5 |
|
value: 45.023 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: BeIR/cqadupstack |
|
name: MTEB CQADupstackProgrammersRetrieval |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 22.491 |
|
- type: map_at_10 |
|
value: 30.434 |
|
- type: map_at_100 |
|
value: 31.611 |
|
- type: map_at_1000 |
|
value: 31.732 |
|
- type: map_at_3 |
|
value: 27.776 |
|
- type: map_at_5 |
|
value: 29.271 |
|
- type: mrr_at_1 |
|
value: 27.74 |
|
- type: mrr_at_10 |
|
value: 34.964 |
|
- type: mrr_at_100 |
|
value: 35.943000000000005 |
|
- type: mrr_at_1000 |
|
value: 36.012 |
|
- type: mrr_at_3 |
|
value: 32.667 |
|
- type: mrr_at_5 |
|
value: 33.975 |
|
- type: ndcg_at_1 |
|
value: 27.74 |
|
- type: ndcg_at_10 |
|
value: 35.32 |
|
- type: ndcg_at_100 |
|
value: 40.812 |
|
- type: ndcg_at_1000 |
|
value: 43.49 |
|
- type: ndcg_at_3 |
|
value: 30.843999999999998 |
|
- type: ndcg_at_5 |
|
value: 32.838 |
|
- type: precision_at_1 |
|
value: 27.74 |
|
- type: precision_at_10 |
|
value: 6.358 |
|
- type: precision_at_100 |
|
value: 1.078 |
|
- type: precision_at_1000 |
|
value: 0.147 |
|
- type: precision_at_3 |
|
value: 14.421999999999999 |
|
- type: precision_at_5 |
|
value: 10.32 |
|
- type: recall_at_1 |
|
value: 22.491 |
|
- type: recall_at_10 |
|
value: 45.659 |
|
- type: recall_at_100 |
|
value: 69.303 |
|
- type: recall_at_1000 |
|
value: 87.849 |
|
- type: recall_at_3 |
|
value: 33.155 |
|
- type: recall_at_5 |
|
value: 38.369 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: BeIR/cqadupstack |
|
name: MTEB CQADupstackRetrieval |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 22.955500000000008 |
|
- type: map_at_10 |
|
value: 30.754000000000005 |
|
- type: map_at_100 |
|
value: 31.85208333333333 |
|
- type: map_at_1000 |
|
value: 31.968416666666666 |
|
- type: map_at_3 |
|
value: 28.35166666666667 |
|
- type: map_at_5 |
|
value: 29.717333333333336 |
|
- type: mrr_at_1 |
|
value: 27.0815 |
|
- type: mrr_at_10 |
|
value: 34.50116666666666 |
|
- type: mrr_at_100 |
|
value: 35.361583333333336 |
|
- type: mrr_at_1000 |
|
value: 35.42583333333334 |
|
- type: mrr_at_3 |
|
value: 32.30499999999999 |
|
- type: mrr_at_5 |
|
value: 33.56175 |
|
- type: ndcg_at_1 |
|
value: 27.0815 |
|
- type: ndcg_at_10 |
|
value: 35.40033333333333 |
|
- type: ndcg_at_100 |
|
value: 40.3485 |
|
- type: ndcg_at_1000 |
|
value: 42.86816666666667 |
|
- type: ndcg_at_3 |
|
value: 31.24325 |
|
- type: ndcg_at_5 |
|
value: 33.21525 |
|
- type: precision_at_1 |
|
value: 27.0815 |
|
- type: precision_at_10 |
|
value: 6.118666666666667 |
|
- type: precision_at_100 |
|
value: 1.0085833333333334 |
|
- type: precision_at_1000 |
|
value: 0.14150000000000001 |
|
- type: precision_at_3 |
|
value: 14.19175 |
|
- type: precision_at_5 |
|
value: 10.064583333333331 |
|
- type: recall_at_1 |
|
value: 22.955500000000008 |
|
- type: recall_at_10 |
|
value: 45.51058333333333 |
|
- type: recall_at_100 |
|
value: 67.49925 |
|
- type: recall_at_1000 |
|
value: 85.24766666666666 |
|
- type: recall_at_3 |
|
value: 33.885 |
|
- type: recall_at_5 |
|
value: 38.99608333333334 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: BeIR/cqadupstack |
|
name: MTEB CQADupstackStatsRetrieval |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 21.371000000000002 |
|
- type: map_at_10 |
|
value: 27.532 |
|
- type: map_at_100 |
|
value: 28.443 |
|
- type: map_at_1000 |
|
value: 28.525 |
|
- type: map_at_3 |
|
value: 25.689 |
|
- type: map_at_5 |
|
value: 26.677 |
|
- type: mrr_at_1 |
|
value: 24.08 |
|
- type: mrr_at_10 |
|
value: 30.128 |
|
- type: mrr_at_100 |
|
value: 30.953999999999997 |
|
- type: mrr_at_1000 |
|
value: 31.022 |
|
- type: mrr_at_3 |
|
value: 28.298000000000002 |
|
- type: mrr_at_5 |
|
value: 29.317 |
|
- type: ndcg_at_1 |
|
value: 24.08 |
|
- type: ndcg_at_10 |
|
value: 31.212 |
|
- type: ndcg_at_100 |
|
value: 35.72 |
|
- type: ndcg_at_1000 |
|
value: 38.061 |
|
- type: ndcg_at_3 |
|
value: 27.705000000000002 |
|
- type: ndcg_at_5 |
|
value: 29.26 |
|
- type: precision_at_1 |
|
value: 24.08 |
|
- type: precision_at_10 |
|
value: 4.8469999999999995 |
|
- type: precision_at_100 |
|
value: 0.753 |
|
- type: precision_at_1000 |
|
value: 0.104 |
|
- type: precision_at_3 |
|
value: 11.759 |
|
- type: precision_at_5 |
|
value: 8.097999999999999 |
|
- type: recall_at_1 |
|
value: 21.371000000000002 |
|
- type: recall_at_10 |
|
value: 40.089000000000006 |
|
- type: recall_at_100 |
|
value: 60.879000000000005 |
|
- type: recall_at_1000 |
|
value: 78.325 |
|
- type: recall_at_3 |
|
value: 30.175 |
|
- type: recall_at_5 |
|
value: 34.168 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: BeIR/cqadupstack |
|
name: MTEB CQADupstackTexRetrieval |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 15.043999999999999 |
|
- type: map_at_10 |
|
value: 20.794 |
|
- type: map_at_100 |
|
value: 21.636 |
|
- type: map_at_1000 |
|
value: 21.753 |
|
- type: map_at_3 |
|
value: 19.006 |
|
- type: map_at_5 |
|
value: 19.994999999999997 |
|
- type: mrr_at_1 |
|
value: 18.066 |
|
- type: mrr_at_10 |
|
value: 24.157999999999998 |
|
- type: mrr_at_100 |
|
value: 24.936 |
|
- type: mrr_at_1000 |
|
value: 25.018 |
|
- type: mrr_at_3 |
|
value: 22.345000000000002 |
|
- type: mrr_at_5 |
|
value: 23.396 |
|
- type: ndcg_at_1 |
|
value: 18.066 |
|
- type: ndcg_at_10 |
|
value: 24.584 |
|
- type: ndcg_at_100 |
|
value: 28.869 |
|
- type: ndcg_at_1000 |
|
value: 31.94 |
|
- type: ndcg_at_3 |
|
value: 21.295 |
|
- type: ndcg_at_5 |
|
value: 22.820999999999998 |
|
- type: precision_at_1 |
|
value: 18.066 |
|
- type: precision_at_10 |
|
value: 4.381 |
|
- type: precision_at_100 |
|
value: 0.754 |
|
- type: precision_at_1000 |
|
value: 0.117 |
|
- type: precision_at_3 |
|
value: 9.956 |
|
- type: precision_at_5 |
|
value: 7.123 |
|
- type: recall_at_1 |
|
value: 15.043999999999999 |
|
- type: recall_at_10 |
|
value: 32.665 |
|
- type: recall_at_100 |
|
value: 52.342 |
|
- type: recall_at_1000 |
|
value: 74.896 |
|
- type: recall_at_3 |
|
value: 23.402 |
|
- type: recall_at_5 |
|
value: 27.397 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: BeIR/cqadupstack |
|
name: MTEB CQADupstackUnixRetrieval |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 22.712 |
|
- type: map_at_10 |
|
value: 28.963 |
|
- type: map_at_100 |
|
value: 29.934 |
|
- type: map_at_1000 |
|
value: 30.049 |
|
- type: map_at_3 |
|
value: 27.086 |
|
- type: map_at_5 |
|
value: 28.163 |
|
- type: mrr_at_1 |
|
value: 26.586 |
|
- type: mrr_at_10 |
|
value: 32.792 |
|
- type: mrr_at_100 |
|
value: 33.692 |
|
- type: mrr_at_1000 |
|
value: 33.767 |
|
- type: mrr_at_3 |
|
value: 30.939 |
|
- type: mrr_at_5 |
|
value: 32.012 |
|
- type: ndcg_at_1 |
|
value: 26.586 |
|
- type: ndcg_at_10 |
|
value: 32.92 |
|
- type: ndcg_at_100 |
|
value: 37.891000000000005 |
|
- type: ndcg_at_1000 |
|
value: 40.647 |
|
- type: ndcg_at_3 |
|
value: 29.465000000000003 |
|
- type: ndcg_at_5 |
|
value: 31.106 |
|
- type: precision_at_1 |
|
value: 26.586 |
|
- type: precision_at_10 |
|
value: 5.177 |
|
- type: precision_at_100 |
|
value: 0.8540000000000001 |
|
- type: precision_at_1000 |
|
value: 0.121 |
|
- type: precision_at_3 |
|
value: 12.903999999999998 |
|
- type: precision_at_5 |
|
value: 8.881 |
|
- type: recall_at_1 |
|
value: 22.712 |
|
- type: recall_at_10 |
|
value: 41.382000000000005 |
|
- type: recall_at_100 |
|
value: 63.866 |
|
- type: recall_at_1000 |
|
value: 83.29299999999999 |
|
- type: recall_at_3 |
|
value: 31.739 |
|
- type: recall_at_5 |
|
value: 35.988 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: BeIR/cqadupstack |
|
name: MTEB CQADupstackWebmastersRetrieval |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 19.64 |
|
- type: map_at_10 |
|
value: 28.432000000000002 |
|
- type: map_at_100 |
|
value: 29.848999999999997 |
|
- type: map_at_1000 |
|
value: 30.072 |
|
- type: map_at_3 |
|
value: 25.862000000000002 |
|
- type: map_at_5 |
|
value: 27.339000000000002 |
|
- type: mrr_at_1 |
|
value: 24.308 |
|
- type: mrr_at_10 |
|
value: 32.475 |
|
- type: mrr_at_100 |
|
value: 33.404 |
|
- type: mrr_at_1000 |
|
value: 33.477000000000004 |
|
- type: mrr_at_3 |
|
value: 30.203999999999997 |
|
- type: mrr_at_5 |
|
value: 31.558000000000003 |
|
- type: ndcg_at_1 |
|
value: 24.308 |
|
- type: ndcg_at_10 |
|
value: 33.79 |
|
- type: ndcg_at_100 |
|
value: 39.113 |
|
- type: ndcg_at_1000 |
|
value: 42.388 |
|
- type: ndcg_at_3 |
|
value: 29.738999999999997 |
|
- type: ndcg_at_5 |
|
value: 31.734 |
|
- type: precision_at_1 |
|
value: 24.308 |
|
- type: precision_at_10 |
|
value: 6.621 |
|
- type: precision_at_100 |
|
value: 1.322 |
|
- type: precision_at_1000 |
|
value: 0.22499999999999998 |
|
- type: precision_at_3 |
|
value: 14.032 |
|
- type: precision_at_5 |
|
value: 10.435 |
|
- type: recall_at_1 |
|
value: 19.64 |
|
- type: recall_at_10 |
|
value: 44.147999999999996 |
|
- type: recall_at_100 |
|
value: 68.31099999999999 |
|
- type: recall_at_1000 |
|
value: 90.022 |
|
- type: recall_at_3 |
|
value: 32.275999999999996 |
|
- type: recall_at_5 |
|
value: 37.717 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: BeIR/cqadupstack |
|
name: MTEB CQADupstackWordpressRetrieval |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 17.443 |
|
- type: map_at_10 |
|
value: 23.45 |
|
- type: map_at_100 |
|
value: 24.41 |
|
- type: map_at_1000 |
|
value: 24.515 |
|
- type: map_at_3 |
|
value: 21.478 |
|
- type: map_at_5 |
|
value: 22.545 |
|
- type: mrr_at_1 |
|
value: 18.854000000000003 |
|
- type: mrr_at_10 |
|
value: 25.174999999999997 |
|
- type: mrr_at_100 |
|
value: 26.099 |
|
- type: mrr_at_1000 |
|
value: 26.179999999999996 |
|
- type: mrr_at_3 |
|
value: 23.352 |
|
- type: mrr_at_5 |
|
value: 24.331 |
|
- type: ndcg_at_1 |
|
value: 18.854000000000003 |
|
- type: ndcg_at_10 |
|
value: 26.99 |
|
- type: ndcg_at_100 |
|
value: 31.823 |
|
- type: ndcg_at_1000 |
|
value: 34.657 |
|
- type: ndcg_at_3 |
|
value: 23.195 |
|
- type: ndcg_at_5 |
|
value: 24.953 |
|
- type: precision_at_1 |
|
value: 18.854000000000003 |
|
- type: precision_at_10 |
|
value: 4.1770000000000005 |
|
- type: precision_at_100 |
|
value: 0.7100000000000001 |
|
- type: precision_at_1000 |
|
value: 0.104 |
|
- type: precision_at_3 |
|
value: 9.797 |
|
- type: precision_at_5 |
|
value: 6.839 |
|
- type: recall_at_1 |
|
value: 17.443 |
|
- type: recall_at_10 |
|
value: 36.22 |
|
- type: recall_at_100 |
|
value: 58.548 |
|
- type: recall_at_1000 |
|
value: 80.104 |
|
- type: recall_at_3 |
|
value: 25.995 |
|
- type: recall_at_5 |
|
value: 30.375999999999998 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: climate-fever |
|
name: MTEB ClimateFEVER |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 10.283000000000001 |
|
- type: map_at_10 |
|
value: 16.121 |
|
- type: map_at_100 |
|
value: 17.818 |
|
- type: map_at_1000 |
|
value: 18.015 |
|
- type: map_at_3 |
|
value: 13.655000000000001 |
|
- type: map_at_5 |
|
value: 14.854999999999999 |
|
- type: mrr_at_1 |
|
value: 22.15 |
|
- type: mrr_at_10 |
|
value: 31.139 |
|
- type: mrr_at_100 |
|
value: 32.336999999999996 |
|
- type: mrr_at_1000 |
|
value: 32.39 |
|
- type: mrr_at_3 |
|
value: 27.861000000000004 |
|
- type: mrr_at_5 |
|
value: 29.754 |
|
- type: ndcg_at_1 |
|
value: 22.15 |
|
- type: ndcg_at_10 |
|
value: 22.852 |
|
- type: ndcg_at_100 |
|
value: 30.233999999999998 |
|
- type: ndcg_at_1000 |
|
value: 34.02 |
|
- type: ndcg_at_3 |
|
value: 18.394 |
|
- type: ndcg_at_5 |
|
value: 19.973 |
|
- type: precision_at_1 |
|
value: 22.15 |
|
- type: precision_at_10 |
|
value: 6.912 |
|
- type: precision_at_100 |
|
value: 1.4829999999999999 |
|
- type: precision_at_1000 |
|
value: 0.218 |
|
- type: precision_at_3 |
|
value: 12.899 |
|
- type: precision_at_5 |
|
value: 10.111 |
|
- type: recall_at_1 |
|
value: 10.283000000000001 |
|
- type: recall_at_10 |
|
value: 27.587 |
|
- type: recall_at_100 |
|
value: 53.273 |
|
- type: recall_at_1000 |
|
value: 74.74499999999999 |
|
- type: recall_at_3 |
|
value: 16.897000000000002 |
|
- type: recall_at_5 |
|
value: 21.084 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: dbpedia-entity |
|
name: MTEB DBPedia |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 9.038 |
|
- type: map_at_10 |
|
value: 20.153 |
|
- type: map_at_100 |
|
value: 28.610999999999997 |
|
- type: map_at_1000 |
|
value: 30.285 |
|
- type: map_at_3 |
|
value: 14.249 |
|
- type: map_at_5 |
|
value: 16.715 |
|
- type: mrr_at_1 |
|
value: 66.75 |
|
- type: mrr_at_10 |
|
value: 74.477 |
|
- type: mrr_at_100 |
|
value: 74.678 |
|
- type: mrr_at_1000 |
|
value: 74.695 |
|
- type: mrr_at_3 |
|
value: 72.625 |
|
- type: mrr_at_5 |
|
value: 73.8 |
|
- type: ndcg_at_1 |
|
value: 55.125 |
|
- type: ndcg_at_10 |
|
value: 41.837999999999994 |
|
- type: ndcg_at_100 |
|
value: 46.182 |
|
- type: ndcg_at_1000 |
|
value: 53.144000000000005 |
|
- type: ndcg_at_3 |
|
value: 46.084 |
|
- type: ndcg_at_5 |
|
value: 43.751 |
|
- type: precision_at_1 |
|
value: 66.75 |
|
- type: precision_at_10 |
|
value: 33.775 |
|
- type: precision_at_100 |
|
value: 10.803 |
|
- type: precision_at_1000 |
|
value: 2.191 |
|
- type: precision_at_3 |
|
value: 49.5 |
|
- type: precision_at_5 |
|
value: 42.4 |
|
- type: recall_at_1 |
|
value: 9.038 |
|
- type: recall_at_10 |
|
value: 25.988 |
|
- type: recall_at_100 |
|
value: 52.158 |
|
- type: recall_at_1000 |
|
value: 74.617 |
|
- type: recall_at_3 |
|
value: 15.675 |
|
- type: recall_at_5 |
|
value: 19.570999999999998 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: fever |
|
name: MTEB FEVER |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 62.551 |
|
- type: map_at_10 |
|
value: 73.124 |
|
- type: map_at_100 |
|
value: 73.432 |
|
- type: map_at_1000 |
|
value: 73.447 |
|
- type: map_at_3 |
|
value: 71.297 |
|
- type: map_at_5 |
|
value: 72.489 |
|
- type: mrr_at_1 |
|
value: 67.23700000000001 |
|
- type: mrr_at_10 |
|
value: 77.438 |
|
- type: mrr_at_100 |
|
value: 77.645 |
|
- type: mrr_at_1000 |
|
value: 77.64999999999999 |
|
- type: mrr_at_3 |
|
value: 75.788 |
|
- type: mrr_at_5 |
|
value: 76.886 |
|
- type: ndcg_at_1 |
|
value: 67.23700000000001 |
|
- type: ndcg_at_10 |
|
value: 78.306 |
|
- type: ndcg_at_100 |
|
value: 79.526 |
|
- type: ndcg_at_1000 |
|
value: 79.825 |
|
- type: ndcg_at_3 |
|
value: 74.961 |
|
- type: ndcg_at_5 |
|
value: 76.91900000000001 |
|
- type: precision_at_1 |
|
value: 67.23700000000001 |
|
- type: precision_at_10 |
|
value: 9.875 |
|
- type: precision_at_100 |
|
value: 1.065 |
|
- type: precision_at_1000 |
|
value: 0.11 |
|
- type: precision_at_3 |
|
value: 29.353 |
|
- type: precision_at_5 |
|
value: 18.749 |
|
- type: recall_at_1 |
|
value: 62.551 |
|
- type: recall_at_10 |
|
value: 90.011 |
|
- type: recall_at_100 |
|
value: 95.06 |
|
- type: recall_at_1000 |
|
value: 97.033 |
|
- type: recall_at_3 |
|
value: 81.081 |
|
- type: recall_at_5 |
|
value: 85.87599999999999 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: fiqa |
|
name: MTEB FiQA2018 |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 17.636 |
|
- type: map_at_10 |
|
value: 28.627000000000002 |
|
- type: map_at_100 |
|
value: 30.262 |
|
- type: map_at_1000 |
|
value: 30.442000000000004 |
|
- type: map_at_3 |
|
value: 25.091 |
|
- type: map_at_5 |
|
value: 27.12 |
|
- type: mrr_at_1 |
|
value: 34.259 |
|
- type: mrr_at_10 |
|
value: 42.733 |
|
- type: mrr_at_100 |
|
value: 43.613 |
|
- type: mrr_at_1000 |
|
value: 43.663000000000004 |
|
- type: mrr_at_3 |
|
value: 40.406 |
|
- type: mrr_at_5 |
|
value: 41.687000000000005 |
|
- type: ndcg_at_1 |
|
value: 34.259 |
|
- type: ndcg_at_10 |
|
value: 35.613 |
|
- type: ndcg_at_100 |
|
value: 42.027 |
|
- type: ndcg_at_1000 |
|
value: 45.336999999999996 |
|
- type: ndcg_at_3 |
|
value: 32.435 |
|
- type: ndcg_at_5 |
|
value: 33.482 |
|
- type: precision_at_1 |
|
value: 34.259 |
|
- type: precision_at_10 |
|
value: 9.66 |
|
- type: precision_at_100 |
|
value: 1.6219999999999999 |
|
- type: precision_at_1000 |
|
value: 0.22300000000000003 |
|
- type: precision_at_3 |
|
value: 21.399 |
|
- type: precision_at_5 |
|
value: 15.741 |
|
- type: recall_at_1 |
|
value: 17.636 |
|
- type: recall_at_10 |
|
value: 41.955999999999996 |
|
- type: recall_at_100 |
|
value: 66.17 |
|
- type: recall_at_1000 |
|
value: 85.79599999999999 |
|
- type: recall_at_3 |
|
value: 29.853 |
|
- type: recall_at_5 |
|
value: 35.18 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: hotpotqa |
|
name: MTEB HotpotQA |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 39.487 |
|
- type: map_at_10 |
|
value: 56.765 |
|
- type: map_at_100 |
|
value: 57.616 |
|
- type: map_at_1000 |
|
value: 57.679 |
|
- type: map_at_3 |
|
value: 53.616 |
|
- type: map_at_5 |
|
value: 55.623999999999995 |
|
- type: mrr_at_1 |
|
value: 78.974 |
|
- type: mrr_at_10 |
|
value: 84.622 |
|
- type: mrr_at_100 |
|
value: 84.776 |
|
- type: mrr_at_1000 |
|
value: 84.783 |
|
- type: mrr_at_3 |
|
value: 83.747 |
|
- type: mrr_at_5 |
|
value: 84.27900000000001 |
|
- type: ndcg_at_1 |
|
value: 78.974 |
|
- type: ndcg_at_10 |
|
value: 66.164 |
|
- type: ndcg_at_100 |
|
value: 69.03099999999999 |
|
- type: ndcg_at_1000 |
|
value: 70.261 |
|
- type: ndcg_at_3 |
|
value: 61.712 |
|
- type: ndcg_at_5 |
|
value: 64.22 |
|
- type: precision_at_1 |
|
value: 78.974 |
|
- type: precision_at_10 |
|
value: 13.520999999999999 |
|
- type: precision_at_100 |
|
value: 1.575 |
|
- type: precision_at_1000 |
|
value: 0.174 |
|
- type: precision_at_3 |
|
value: 38.501000000000005 |
|
- type: precision_at_5 |
|
value: 25.083 |
|
- type: recall_at_1 |
|
value: 39.487 |
|
- type: recall_at_10 |
|
value: 67.60300000000001 |
|
- type: recall_at_100 |
|
value: 78.744 |
|
- type: recall_at_1000 |
|
value: 86.914 |
|
- type: recall_at_3 |
|
value: 57.752 |
|
- type: recall_at_5 |
|
value: 62.708 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: msmarco |
|
name: MTEB MSMARCO |
|
config: default |
|
split: dev |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 24.224999999999998 |
|
- type: map_at_10 |
|
value: 37.791000000000004 |
|
- type: map_at_100 |
|
value: 38.899 |
|
- type: map_at_1000 |
|
value: 38.937 |
|
- type: map_at_3 |
|
value: 33.584 |
|
- type: map_at_5 |
|
value: 36.142 |
|
- type: mrr_at_1 |
|
value: 24.871 |
|
- type: mrr_at_10 |
|
value: 38.361000000000004 |
|
- type: mrr_at_100 |
|
value: 39.394 |
|
- type: mrr_at_1000 |
|
value: 39.427 |
|
- type: mrr_at_3 |
|
value: 34.224 |
|
- type: mrr_at_5 |
|
value: 36.767 |
|
- type: ndcg_at_1 |
|
value: 24.871 |
|
- type: ndcg_at_10 |
|
value: 45.231 |
|
- type: ndcg_at_100 |
|
value: 50.42100000000001 |
|
- type: ndcg_at_1000 |
|
value: 51.329 |
|
- type: ndcg_at_3 |
|
value: 36.77 |
|
- type: ndcg_at_5 |
|
value: 41.33 |
|
- type: precision_at_1 |
|
value: 24.871 |
|
- type: precision_at_10 |
|
value: 7.124999999999999 |
|
- type: precision_at_100 |
|
value: 0.971 |
|
- type: precision_at_1000 |
|
value: 0.105 |
|
- type: precision_at_3 |
|
value: 15.659 |
|
- type: precision_at_5 |
|
value: 11.708 |
|
- type: recall_at_1 |
|
value: 24.224999999999998 |
|
- type: recall_at_10 |
|
value: 68.081 |
|
- type: recall_at_100 |
|
value: 91.818 |
|
- type: recall_at_1000 |
|
value: 98.65 |
|
- type: recall_at_3 |
|
value: 45.355000000000004 |
|
- type: recall_at_5 |
|
value: 56.26 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: nfcorpus |
|
name: MTEB NFCorpus |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 5.904 |
|
- type: map_at_10 |
|
value: 12.784 |
|
- type: map_at_100 |
|
value: 15.628 |
|
- type: map_at_1000 |
|
value: 17.006 |
|
- type: map_at_3 |
|
value: 9.695 |
|
- type: map_at_5 |
|
value: 10.961 |
|
- type: mrr_at_1 |
|
value: 46.44 |
|
- type: mrr_at_10 |
|
value: 54.106 |
|
- type: mrr_at_100 |
|
value: 54.81700000000001 |
|
- type: mrr_at_1000 |
|
value: 54.858 |
|
- type: mrr_at_3 |
|
value: 52.837999999999994 |
|
- type: mrr_at_5 |
|
value: 53.627 |
|
- type: ndcg_at_1 |
|
value: 44.737 |
|
- type: ndcg_at_10 |
|
value: 33.967999999999996 |
|
- type: ndcg_at_100 |
|
value: 30.451 |
|
- type: ndcg_at_1000 |
|
value: 39.151 |
|
- type: ndcg_at_3 |
|
value: 39.871 |
|
- type: ndcg_at_5 |
|
value: 37.138 |
|
- type: precision_at_1 |
|
value: 46.44 |
|
- type: precision_at_10 |
|
value: 24.582 |
|
- type: precision_at_100 |
|
value: 7.715 |
|
- type: precision_at_1000 |
|
value: 2.0500000000000003 |
|
- type: precision_at_3 |
|
value: 37.461 |
|
- type: precision_at_5 |
|
value: 31.517 |
|
- type: recall_at_1 |
|
value: 5.904 |
|
- type: recall_at_10 |
|
value: 16.522000000000002 |
|
- type: recall_at_100 |
|
value: 29.413 |
|
- type: recall_at_1000 |
|
value: 61.611000000000004 |
|
- type: recall_at_3 |
|
value: 10.649000000000001 |
|
- type: recall_at_5 |
|
value: 12.642999999999999 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: nq |
|
name: MTEB NQ |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 31.561 |
|
- type: map_at_10 |
|
value: 46.406 |
|
- type: map_at_100 |
|
value: 47.499 |
|
- type: map_at_1000 |
|
value: 47.526 |
|
- type: map_at_3 |
|
value: 42.26 |
|
- type: map_at_5 |
|
value: 44.724000000000004 |
|
- type: mrr_at_1 |
|
value: 35.168 |
|
- type: mrr_at_10 |
|
value: 48.914 |
|
- type: mrr_at_100 |
|
value: 49.727 |
|
- type: mrr_at_1000 |
|
value: 49.744 |
|
- type: mrr_at_3 |
|
value: 45.418 |
|
- type: mrr_at_5 |
|
value: 47.53 |
|
- type: ndcg_at_1 |
|
value: 35.138999999999996 |
|
- type: ndcg_at_10 |
|
value: 53.943 |
|
- type: ndcg_at_100 |
|
value: 58.50300000000001 |
|
- type: ndcg_at_1000 |
|
value: 59.144 |
|
- type: ndcg_at_3 |
|
value: 46.135999999999996 |
|
- type: ndcg_at_5 |
|
value: 50.227999999999994 |
|
- type: precision_at_1 |
|
value: 35.138999999999996 |
|
- type: precision_at_10 |
|
value: 8.812000000000001 |
|
- type: precision_at_100 |
|
value: 1.138 |
|
- type: precision_at_1000 |
|
value: 0.12 |
|
- type: precision_at_3 |
|
value: 20.867 |
|
- type: precision_at_5 |
|
value: 14.878 |
|
- type: recall_at_1 |
|
value: 31.561 |
|
- type: recall_at_10 |
|
value: 74.343 |
|
- type: recall_at_100 |
|
value: 93.975 |
|
- type: recall_at_1000 |
|
value: 98.75699999999999 |
|
- type: recall_at_3 |
|
value: 54.169 |
|
- type: recall_at_5 |
|
value: 63.56 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: quora |
|
name: MTEB QuoraRetrieval |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 69.753 |
|
- type: map_at_10 |
|
value: 83.56400000000001 |
|
- type: map_at_100 |
|
value: 84.19200000000001 |
|
- type: map_at_1000 |
|
value: 84.211 |
|
- type: map_at_3 |
|
value: 80.568 |
|
- type: map_at_5 |
|
value: 82.44500000000001 |
|
- type: mrr_at_1 |
|
value: 79.99000000000001 |
|
- type: mrr_at_10 |
|
value: 86.542 |
|
- type: mrr_at_100 |
|
value: 86.655 |
|
- type: mrr_at_1000 |
|
value: 86.656 |
|
- type: mrr_at_3 |
|
value: 85.505 |
|
- type: mrr_at_5 |
|
value: 86.21 |
|
- type: ndcg_at_1 |
|
value: 79.99000000000001 |
|
- type: ndcg_at_10 |
|
value: 87.449 |
|
- type: ndcg_at_100 |
|
value: 88.739 |
|
- type: ndcg_at_1000 |
|
value: 88.87 |
|
- type: ndcg_at_3 |
|
value: 84.418 |
|
- type: ndcg_at_5 |
|
value: 86.09599999999999 |
|
- type: precision_at_1 |
|
value: 79.99000000000001 |
|
- type: precision_at_10 |
|
value: 13.236999999999998 |
|
- type: precision_at_100 |
|
value: 1.516 |
|
- type: precision_at_1000 |
|
value: 0.156 |
|
- type: precision_at_3 |
|
value: 36.736999999999995 |
|
- type: precision_at_5 |
|
value: 24.227999999999998 |
|
- type: recall_at_1 |
|
value: 69.753 |
|
- type: recall_at_10 |
|
value: 94.967 |
|
- type: recall_at_100 |
|
value: 99.378 |
|
- type: recall_at_1000 |
|
value: 99.953 |
|
- type: recall_at_3 |
|
value: 86.408 |
|
- type: recall_at_5 |
|
value: 91.03 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: scidocs |
|
name: MTEB SCIDOCS |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 3.8080000000000003 |
|
- type: map_at_10 |
|
value: 9.222 |
|
- type: map_at_100 |
|
value: 10.779 |
|
- type: map_at_1000 |
|
value: 11.027000000000001 |
|
- type: map_at_3 |
|
value: 6.729 |
|
- type: map_at_5 |
|
value: 7.872999999999999 |
|
- type: mrr_at_1 |
|
value: 18.7 |
|
- type: mrr_at_10 |
|
value: 28.084999999999997 |
|
- type: mrr_at_100 |
|
value: 29.134999999999998 |
|
- type: mrr_at_1000 |
|
value: 29.214000000000002 |
|
- type: mrr_at_3 |
|
value: 24.917 |
|
- type: mrr_at_5 |
|
value: 26.651999999999997 |
|
- type: ndcg_at_1 |
|
value: 18.7 |
|
- type: ndcg_at_10 |
|
value: 15.969 |
|
- type: ndcg_at_100 |
|
value: 22.535 |
|
- type: ndcg_at_1000 |
|
value: 27.337 |
|
- type: ndcg_at_3 |
|
value: 15.112 |
|
- type: ndcg_at_5 |
|
value: 13.089 |
|
- type: precision_at_1 |
|
value: 18.7 |
|
- type: precision_at_10 |
|
value: 8.32 |
|
- type: precision_at_100 |
|
value: 1.786 |
|
- type: precision_at_1000 |
|
value: 0.293 |
|
- type: precision_at_3 |
|
value: 14.099999999999998 |
|
- type: precision_at_5 |
|
value: 11.42 |
|
- type: recall_at_1 |
|
value: 3.8080000000000003 |
|
- type: recall_at_10 |
|
value: 16.872 |
|
- type: recall_at_100 |
|
value: 36.235 |
|
- type: recall_at_1000 |
|
value: 59.587 |
|
- type: recall_at_3 |
|
value: 8.583 |
|
- type: recall_at_5 |
|
value: 11.562999999999999 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: scifact |
|
name: MTEB SciFact |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 53.994 |
|
- type: map_at_10 |
|
value: 63.56 |
|
- type: map_at_100 |
|
value: 64.247 |
|
- type: map_at_1000 |
|
value: 64.275 |
|
- type: map_at_3 |
|
value: 61.23499999999999 |
|
- type: map_at_5 |
|
value: 62.638000000000005 |
|
- type: mrr_at_1 |
|
value: 57.333 |
|
- type: mrr_at_10 |
|
value: 65.23299999999999 |
|
- type: mrr_at_100 |
|
value: 65.762 |
|
- type: mrr_at_1000 |
|
value: 65.78699999999999 |
|
- type: mrr_at_3 |
|
value: 63.556000000000004 |
|
- type: mrr_at_5 |
|
value: 64.572 |
|
- type: ndcg_at_1 |
|
value: 57.333 |
|
- type: ndcg_at_10 |
|
value: 67.88300000000001 |
|
- type: ndcg_at_100 |
|
value: 70.99 |
|
- type: ndcg_at_1000 |
|
value: 71.66 |
|
- type: ndcg_at_3 |
|
value: 64.16 |
|
- type: ndcg_at_5 |
|
value: 66.042 |
|
- type: precision_at_1 |
|
value: 57.333 |
|
- type: precision_at_10 |
|
value: 8.967 |
|
- type: precision_at_100 |
|
value: 1.06 |
|
- type: precision_at_1000 |
|
value: 0.11199999999999999 |
|
- type: precision_at_3 |
|
value: 25.222 |
|
- type: precision_at_5 |
|
value: 16.467000000000002 |
|
- type: recall_at_1 |
|
value: 53.994 |
|
- type: recall_at_10 |
|
value: 79.289 |
|
- type: recall_at_100 |
|
value: 93.533 |
|
- type: recall_at_1000 |
|
value: 98.667 |
|
- type: recall_at_3 |
|
value: 69.267 |
|
- type: recall_at_5 |
|
value: 74.128 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: trec-covid |
|
name: MTEB TRECCOVID |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 0.212 |
|
- type: map_at_10 |
|
value: 1.925 |
|
- type: map_at_100 |
|
value: 9.235 |
|
- type: map_at_1000 |
|
value: 22.111 |
|
- type: map_at_3 |
|
value: 0.626 |
|
- type: map_at_5 |
|
value: 1.031 |
|
- type: mrr_at_1 |
|
value: 82.0 |
|
- type: mrr_at_10 |
|
value: 90.5 |
|
- type: mrr_at_100 |
|
value: 90.5 |
|
- type: mrr_at_1000 |
|
value: 90.5 |
|
- type: mrr_at_3 |
|
value: 90.0 |
|
- type: mrr_at_5 |
|
value: 90.5 |
|
- type: ndcg_at_1 |
|
value: 75.0 |
|
- type: ndcg_at_10 |
|
value: 75.851 |
|
- type: ndcg_at_100 |
|
value: 53.190000000000005 |
|
- type: ndcg_at_1000 |
|
value: 45.507999999999996 |
|
- type: ndcg_at_3 |
|
value: 80.19500000000001 |
|
- type: ndcg_at_5 |
|
value: 78.448 |
|
- type: precision_at_1 |
|
value: 82.0 |
|
- type: precision_at_10 |
|
value: 82.6 |
|
- type: precision_at_100 |
|
value: 54.48 |
|
- type: precision_at_1000 |
|
value: 20.785999999999998 |
|
- type: precision_at_3 |
|
value: 86.667 |
|
- type: precision_at_5 |
|
value: 85.2 |
|
- type: recall_at_1 |
|
value: 0.212 |
|
- type: recall_at_10 |
|
value: 2.13 |
|
- type: recall_at_100 |
|
value: 12.152000000000001 |
|
- type: recall_at_1000 |
|
value: 42.403 |
|
- type: recall_at_3 |
|
value: 0.6689999999999999 |
|
- type: recall_at_5 |
|
value: 1.121 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: webis-touche2020 |
|
name: MTEB Touche2020 |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 2.701 |
|
- type: map_at_10 |
|
value: 10.488999999999999 |
|
- type: map_at_100 |
|
value: 17.258000000000003 |
|
- type: map_at_1000 |
|
value: 18.797 |
|
- type: map_at_3 |
|
value: 5.563 |
|
- type: map_at_5 |
|
value: 7.268 |
|
- type: mrr_at_1 |
|
value: 30.612000000000002 |
|
- type: mrr_at_10 |
|
value: 48.197 |
|
- type: mrr_at_100 |
|
value: 48.762 |
|
- type: mrr_at_1000 |
|
value: 48.762 |
|
- type: mrr_at_3 |
|
value: 44.218 |
|
- type: mrr_at_5 |
|
value: 46.666999999999994 |
|
- type: ndcg_at_1 |
|
value: 28.571 |
|
- type: ndcg_at_10 |
|
value: 26.512 |
|
- type: ndcg_at_100 |
|
value: 38.356 |
|
- type: ndcg_at_1000 |
|
value: 49.57 |
|
- type: ndcg_at_3 |
|
value: 27.704 |
|
- type: ndcg_at_5 |
|
value: 27.342 |
|
- type: precision_at_1 |
|
value: 30.612000000000002 |
|
- type: precision_at_10 |
|
value: 24.285999999999998 |
|
- type: precision_at_100 |
|
value: 8.0 |
|
- type: precision_at_1000 |
|
value: 1.541 |
|
- type: precision_at_3 |
|
value: 29.252 |
|
- type: precision_at_5 |
|
value: 27.346999999999998 |
|
- type: recall_at_1 |
|
value: 2.701 |
|
- type: recall_at_10 |
|
value: 17.197000000000003 |
|
- type: recall_at_100 |
|
value: 49.061 |
|
- type: recall_at_1000 |
|
value: 82.82300000000001 |
|
- type: recall_at_3 |
|
value: 6.687 |
|
- type: recall_at_5 |
|
value: 9.868 |
|
--- |
|
DRAGON+ is a BERT-base sized dense retriever initialized from [RetroMAE](https://huggingface.co/Shitao/RetroMAE) and further trained on the data augmented from MS MARCO corpus, following the approach described in [How to Train Your DRAGON: |
|
Diverse Augmentation Towards Generalizable Dense Retrieval](https://arxiv.org/abs/2302.07452). |
|
|
|
<p align="center"> |
|
<img src="https://raw.githubusercontent.com/facebookresearch/dpr-scale/main/dragon/images/teaser.png" width="600"> |
|
</p> |
|
|
|
The associated GitHub repository is available here https://github.com/facebookresearch/dpr-scale/tree/main/dragon. We use asymmetric dual encoder, with two distinctly parameterized encoders. The following models are also available: |
|
|
|
Model | Initialization | MARCO Dev | BEIR | Query Encoder Path | Context Encoder Path |
|
|---|---|---|---|---|--- |
|
DRAGON+ | Shitao/RetroMAE| 39.0 | 47.4 | [facebook/dragon-plus-query-encoder](https://huggingface.co/facebook/dragon-plus-query-encoder) | [facebook/dragon-plus-context-encoder](https://huggingface.co/facebook/dragon-plus-context-encoder) |
|
DRAGON-RoBERTa | RoBERTa-base | 39.4 | 47.2 | [facebook/dragon-roberta-query-encoder](https://huggingface.co/facebook/dragon-roberta-query-encoder) | [facebook/dragon-roberta-context-encoder](https://huggingface.co/facebook/dragon-roberta-context-encoder) |
|
|
|
## Usage (HuggingFace Transformers) |
|
Using the model directly available in HuggingFace transformers . |
|
|
|
```python |
|
import torch |
|
from transformers import AutoTokenizer, AutoModel |
|
tokenizer = AutoTokenizer.from_pretrained('facebook/dragon-plus-query-encoder') |
|
query_encoder = AutoModel.from_pretrained('facebook/dragon-plus-query-encoder') |
|
context_encoder = AutoModel.from_pretrained('facebook/dragon-plus-context-encoder') |
|
|
|
# We use msmarco query and passages as an example |
|
query = "Where was Marie Curie born?" |
|
contexts = [ |
|
"Maria Sklodowska, later known as Marie Curie, was born on November 7, 1867.", |
|
"Born in Paris on 15 May 1859, Pierre Curie was the son of Eugène Curie, a doctor of French Catholic origin from Alsace." |
|
] |
|
# Apply tokenizer |
|
query_input = tokenizer(query, return_tensors='pt') |
|
ctx_input = tokenizer(contexts, padding=True, truncation=True, return_tensors='pt') |
|
# Compute embeddings: take the last-layer hidden state of the [CLS] token |
|
query_emb = query_encoder(**query_input).last_hidden_state[:, 0, :] |
|
ctx_emb = context_encoder(**ctx_input).last_hidden_state[:, 0, :] |
|
# Compute similarity scores using dot product |
|
score1 = query_emb @ ctx_emb[0] # 396.5625 |
|
score2 = query_emb @ ctx_emb[1] # 393.8340 |
|
``` |
