File size: 53,851 Bytes
46e2a6c e8b670b 46e2a6c |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 511 512 513 514 515 516 517 518 519 520 521 522 523 524 525 526 527 528 529 530 531 532 533 534 535 536 537 538 539 540 541 542 543 544 545 546 547 548 549 550 551 552 553 554 555 556 557 558 559 560 561 562 563 564 565 566 567 568 569 570 571 572 573 574 575 576 577 578 579 580 581 582 583 584 585 586 587 588 589 590 591 592 593 594 595 596 597 598 599 600 601 602 603 604 605 606 607 608 609 610 611 612 613 614 615 616 617 618 619 620 621 622 623 624 625 626 627 628 629 630 631 632 633 634 635 636 637 638 639 640 641 642 643 644 645 646 647 648 649 650 651 652 653 654 655 656 657 658 659 660 661 662 663 664 665 666 667 668 669 670 671 672 673 674 675 676 677 678 679 680 681 682 683 684 685 686 687 688 689 690 691 692 693 694 695 696 697 698 699 700 701 702 703 704 705 706 707 708 709 710 711 712 713 714 715 716 717 718 719 720 721 722 723 724 725 726 727 728 729 730 731 732 733 734 735 736 737 738 739 740 741 742 743 744 745 746 747 748 749 750 751 752 753 754 755 756 757 758 759 760 761 762 763 764 765 766 767 768 769 770 771 772 773 774 775 776 777 778 779 780 781 782 783 784 785 786 787 788 789 790 791 792 793 794 795 796 797 798 799 800 801 802 803 804 805 806 807 808 809 810 811 812 813 814 815 816 817 818 819 820 821 822 |
---
language:
- en
tags:
- sentence-transformers
- sentence-similarity
- feature-extraction
- generated_from_trainer
- dataset_size:100231
- loss:MultipleNegativesRankingLoss
base_model: answerdotai/ModernBERT-base
widget:
- source_sentence: who led the army that defeated the aztecs
sentences:
- Spanish conquest of the Aztec Empire The Spanish conquest of the Aztec Empire,
or the Spanish-Aztec War (1519-21)[3] was one of the most significant and complex
events in world history. There are multiple sixteenth-century narratives of the
events by Spanish conquerors, their indigenous allies, and the defeated Aztecs.
It was not solely a contest between a small contingent of Spaniards defeating
the Aztec Empire, but rather the creation of a coalition of Spanish invaders with
tributaries to the Aztecs, and most especially the Aztecs' indigenous enemies
and rivals. They combined forces to defeat the Mexica of Tenochtitlan over a two-year
period. For the Spanish, the expedition to Mexico was part of a project of Spanish
colonization of the New World after twenty-five years of permanent Spanish settlement
and further exploration in the Caribbean. The Spanish made landfall in Mexico
in 1517. A Spanish settler in Cuba, Hernán Cortés, led an expedition (entrada)
to Mexico, landing in February 1519, following an earlier expedition led by Juan
de Grijalva to Yucatán in 1517. Two years later Cortés and his retinue set sail,
thus beginning the expedition of exploration and conquest.[4] The Spanish campaign
against the Aztec Empire had its final victory on August 13, 1521, when a coalition
army of Spanish forces and native Tlaxcalan warriors led by Cortés and Xicotencatl
the Younger captured the emperor Cuauhtemoc and Tenochtitlan, the capital of the
Aztec Empire. The fall of Tenochtitlan marks the beginning of Spanish rule in
central Mexico, and they established their capital of Mexico City on the ruins
of Tenochtitlan.
- The Girl with All the Gifts Justineau awakens in the Rosalind Franklin. Melanie
leads her to a group of intelligent hungries, to whom Justineau, wearing an environmental
protection suit, starts teaching the alphabet.
- 'Wendy Makkena In 1992 she had a supporting role in the movie Sister Act as the
shy but talented singing nun Sister Mary Robert, a role she reprised in Sister
Act 2: Back in the Habit the following year. She appeared in various other television
roles until 1997, when she starred in Air Bud, followed by the independent film
Finding North. She continued appearing on television shows such as The Job, Oliver
Beene, and Listen Up![citation needed]'
- source_sentence: who went to the most nba finals in a row
sentences:
- List of NBA franchise post-season streaks The San Antonio Spurs hold the longest
active consecutive playoff appearances with 21 appearances, starting in the 1998
NBA Playoffs (also the longest active playoff streak in any major North American
sports league as of 2017). The Spurs have won five NBA championships during the
streak. The Philadelphia 76ers (formerly known as Syracuse Nationals) hold the
all-time record for consecutive playoff appearances with 22 straight appearances
between 1950 and 1971. The 76ers won two NBA championships during their streak.
The Boston Celtics hold the longest consecutive NBA Finals appearance streak with
ten appearances between 1957 and 1966. During the streak, the Celtics won eight
consecutive NBA championships—also an NBA record.
- Dear Dumb Diary Dear Dumb Diary is a series of children's novels by Jim Benton.
Each book is written in the first person view of a middle school girl named Jamie
Kelly. The series is published by Scholastic in English and Random House in Korean.
Film rights to the series have been optioned by the Gotham Group.[2]
- Voting rights in the United States Eligibility to vote in the United States is
established both through the federal constitution and by state law. Several constitutional
amendments (the 15th, 19th, and 26th specifically) require that voting rights
cannot be abridged on account of race, color, previous condition of servitude,
sex, or age for those above 18; the constitution as originally written did not
establish any such rights during 1787–1870. In the absence of a specific federal
law or constitutional provision, each state is given considerable discretion to
establish qualifications for suffrage and candidacy within its own respective
jurisdiction; in addition, states and lower level jurisdictions establish election
systems, such as at-large or single member district elections for county councils
or school boards.
- source_sentence: who did the vocals on mcdonald's jingle i'm loving it
sentences:
- I'm Lovin' It (song) "I'm Lovin' It" is a song recorded by American singer-songwriter
Justin Timberlake. It was written by Pusha T and produced by The Neptunes.
- Vallabhbhai Patel As the first Home Minister and Deputy Prime Minister of India,
Patel organised relief efforts for refugees fleeing from Punjab and Delhi and
worked to restore peace across the nation. He led the task of forging a united
India, successfully integrating into the newly independent nation those British
colonial provinces that had been "allocated" to India. Besides those provinces
that had been under direct British rule, approximately 565 self-governing princely
states had been released from British suzerainty by the Indian Independence Act
of 1947. Employing frank diplomacy with the expressed option to deploy military
force, Patel persuaded almost every princely state to accede to India. His commitment
to national integration in the newly independent country was total and uncompromising,
earning him the sobriquet "Iron Man of India".[3] He is also affectionately remembered
as the "Patron saint of India's civil servants" for having established the modern
all-India services system. He is also called the Unifier of India.[4]
- National debt of the United States As of July 31, 2018, debt held by the public
was $15.6 trillion and intragovernmental holdings were $5.7 trillion, for a total
or "National Debt" of $21.3 trillion.[5] Debt held by the public was approximately
77% of GDP in 2017, ranked 43rd highest out of 207 countries.[6] The Congressional
Budget Office forecast in April 2018 that the ratio will rise to nearly 100% by
2028, perhaps higher if current policies are extended beyond their scheduled expiration
date.[7] As of December 2017, $6.3 trillion or approximately 45% of the debt held
by the public was owned by foreign investors, the largest being China (about $1.18
trillion) then Japan (about $1.06 trillion).[8]
- source_sentence: who is the actress of harley quinn in suicide squad
sentences:
- Tariffs in United States history Tariffs were the main source of revenue for the
federal government from 1789 to 1914. During this period, there was vigorous debate
between the various political parties over the setting of tariff rates. In general
Democrats favored a tariff that would pay the cost of government, but no higher.
Whigs and Republicans favored higher tariffs to protect and encourage American
industry and industrial workers. Since the early 20th century, however, U.S. tariffs
have been very low and have been much less a matter of partisan debate.
- The Rolling Stones The Rolling Stones are an English rock band formed in London,
England in 1962. The first stable line-up consisted of Brian Jones (guitar, harmonica),
Mick Jagger (lead vocals), Keith Richards (guitar, backing vocals), Bill Wyman
(bass), Charlie Watts (drums), and Ian Stewart (piano). Stewart was removed from
the official line-up in 1963 but continued as a touring member until his death
in 1985. Jones left the band less than a month prior to his death in 1969, having
already been replaced by Mick Taylor, who remained until 1974. After Taylor left
the band, Ronnie Wood took his place in 1975 and has been on guitar in tandem
with Richards ever since. Following Wyman's departure in 1993, Darryl Jones joined
as their touring bassist. Touring keyboardists for the band have been Nicky Hopkins
(1967–1982), Ian McLagan (1978–1981), Billy Preston (through the mid-1970s) and
Chuck Leavell (1982–present). The band was first led by Brian Jones, but after
developing into the band's songwriters, Jagger and Richards assumed leadership
while Jones dealt with legal and personal troubles.
- Margot Robbie After moving to the United States, Robbie starred in the short-lived
ABC drama series Pan Am (2011–2012). In 2013, she made her big screen debut in
Richard Curtis's romantic comedy-drama film About Time and co-starred in Martin
Scorsese's biographical black comedy The Wolf of Wall Street. In 2015, Robbie
co-starred in the romantic comedy-drama film Focus, appeared in the romantic World
War II drama film Suite Française and starred in the science fiction film Z for
Zachariah. That same year, she played herself in The Big Short. In 2016, she portrayed
Jane Porter in the action-adventure film The Legend of Tarzan and Harley Quinn
in the superhero film Suicide Squad. She appeared on Time magazine's "The Most
Influential People of 2017" list.[4]
- source_sentence: what is meaning of am and pm in time
sentences:
- America's Got Talent America's Got Talent (often abbreviated as AGT) is a televised
American talent show competition, broadcast on the NBC television network. It
is part of the global Got Talent franchise created by Simon Cowell, and is produced
by Fremantle North America and SYCOtv, with distribution done by Fremantle. Since
its premiere in June 2006, each season is run during the network's summer schedule,
with the show having featured various hosts - it is currently hosted by Tyra Banks,
since 2017.[2] It is the first global edition of the franchise, after plans for
a British edition in 2005 were suspended, following a dispute between Paul O'Grady,
the planned host, and the British broadcaster ITV; production of this edition
later resumed in 2007.[3]
- Times Square Times Square is a major commercial intersection, tourist destination,
entertainment center and neighborhood in the Midtown Manhattan section of New
York City at the junction of Broadway and Seventh Avenue. It stretches from West
42nd to West 47th Streets.[1] Brightly adorned with billboards and advertisements,
Times Square is sometimes referred to as "The Crossroads of the World",[2] "The
Center of the Universe",[3] "the heart of The Great White Way",[4][5][6] and the
"heart of the world".[7] One of the world's busiest pedestrian areas,[8] it is
also the hub of the Broadway Theater District[9] and a major center of the world's
entertainment industry.[10] Times Square is one of the world's most visited tourist
attractions, drawing an estimated 50 million visitors annually.[11] Approximately
330,000 people pass through Times Square daily,[12] many of them tourists,[13]
while over 460,000 pedestrians walk through Times Square on its busiest days.[7]
- '12-hour clock The 12-hour clock is a time convention in which the 24 hours of
the day are divided into two periods:[1] a.m. (from the Latin, ante meridiem,
meaning before midday) and p.m. (post meridiem, meaning past midday).[2] Each
period consists of 12 hours numbered: 12 (acting as zero),[3] 1, 2, 3, 4, 5, 6,
7, 8, 9, 10, and 11. The 24 hour/day cycle starts at 12 midnight (often indicated
as 12 a.m.), runs through 12 noon (often indicated as 12 p.m.), and continues
to the midnight at the end of the day. The 12-hour clock was developed over time
from the mid-second millennium BC to the 16th century AD.'
datasets:
- sentence-transformers/natural-questions
pipeline_tag: sentence-similarity
library_name: sentence-transformers
metrics:
- cosine_accuracy@1
- cosine_accuracy@3
- cosine_accuracy@5
- cosine_accuracy@10
- cosine_precision@1
- cosine_precision@3
- cosine_precision@5
- cosine_precision@10
- cosine_recall@1
- cosine_recall@3
- cosine_recall@5
- cosine_recall@10
- cosine_ndcg@10
- cosine_mrr@10
- cosine_map@100
model-index:
- name: SentenceTransformer based on answerdotai/ModernBERT-base
results:
- task:
type: information-retrieval
name: Information Retrieval
dataset:
name: NanoMSMARCO
type: NanoMSMARCO
metrics:
- type: cosine_accuracy@1
value: 0.24
name: Cosine Accuracy@1
- type: cosine_accuracy@3
value: 0.44
name: Cosine Accuracy@3
- type: cosine_accuracy@5
value: 0.58
name: Cosine Accuracy@5
- type: cosine_accuracy@10
value: 0.72
name: Cosine Accuracy@10
- type: cosine_precision@1
value: 0.24
name: Cosine Precision@1
- type: cosine_precision@3
value: 0.14666666666666664
name: Cosine Precision@3
- type: cosine_precision@5
value: 0.11599999999999999
name: Cosine Precision@5
- type: cosine_precision@10
value: 0.07200000000000001
name: Cosine Precision@10
- type: cosine_recall@1
value: 0.24
name: Cosine Recall@1
- type: cosine_recall@3
value: 0.44
name: Cosine Recall@3
- type: cosine_recall@5
value: 0.58
name: Cosine Recall@5
- type: cosine_recall@10
value: 0.72
name: Cosine Recall@10
- type: cosine_ndcg@10
value: 0.4602960319216384
name: Cosine Ndcg@10
- type: cosine_mrr@10
value: 0.37971428571428567
name: Cosine Mrr@10
- type: cosine_map@100
value: 0.39452525516732045
name: Cosine Map@100
- task:
type: information-retrieval
name: Information Retrieval
dataset:
name: NanoHotpotQA
type: NanoHotpotQA
metrics:
- type: cosine_accuracy@1
value: 0.54
name: Cosine Accuracy@1
- type: cosine_accuracy@3
value: 0.62
name: Cosine Accuracy@3
- type: cosine_accuracy@5
value: 0.64
name: Cosine Accuracy@5
- type: cosine_accuracy@10
value: 0.74
name: Cosine Accuracy@10
- type: cosine_precision@1
value: 0.54
name: Cosine Precision@1
- type: cosine_precision@3
value: 0.28
name: Cosine Precision@3
- type: cosine_precision@5
value: 0.176
name: Cosine Precision@5
- type: cosine_precision@10
value: 0.11
name: Cosine Precision@10
- type: cosine_recall@1
value: 0.27
name: Cosine Recall@1
- type: cosine_recall@3
value: 0.42
name: Cosine Recall@3
- type: cosine_recall@5
value: 0.44
name: Cosine Recall@5
- type: cosine_recall@10
value: 0.55
name: Cosine Recall@10
- type: cosine_ndcg@10
value: 0.49588567362388986
name: Cosine Ndcg@10
- type: cosine_mrr@10
value: 0.5962142857142857
name: Cosine Mrr@10
- type: cosine_map@100
value: 0.43201211644685455
name: Cosine Map@100
- task:
type: nano-beir
name: Nano BEIR
dataset:
name: NanoBEIR mean
type: NanoBEIR_mean
metrics:
- type: cosine_accuracy@1
value: 0.39
name: Cosine Accuracy@1
- type: cosine_accuracy@3
value: 0.53
name: Cosine Accuracy@3
- type: cosine_accuracy@5
value: 0.61
name: Cosine Accuracy@5
- type: cosine_accuracy@10
value: 0.73
name: Cosine Accuracy@10
- type: cosine_precision@1
value: 0.39
name: Cosine Precision@1
- type: cosine_precision@3
value: 0.21333333333333332
name: Cosine Precision@3
- type: cosine_precision@5
value: 0.146
name: Cosine Precision@5
- type: cosine_precision@10
value: 0.091
name: Cosine Precision@10
- type: cosine_recall@1
value: 0.255
name: Cosine Recall@1
- type: cosine_recall@3
value: 0.43
name: Cosine Recall@3
- type: cosine_recall@5
value: 0.51
name: Cosine Recall@5
- type: cosine_recall@10
value: 0.635
name: Cosine Recall@10
- type: cosine_ndcg@10
value: 0.47809085277276414
name: Cosine Ndcg@10
- type: cosine_mrr@10
value: 0.4879642857142857
name: Cosine Mrr@10
- type: cosine_map@100
value: 0.41326868580708753
name: Cosine Map@100
---
# SentenceTransformer based on answerdotai/ModernBERT-base
This is a [sentence-transformers](https://www.SBERT.net) model finetuned from [answerdotai/ModernBERT-base](https://huggingface.co/answerdotai/ModernBERT-base) on the [natural-questions](https://huggingface.co/datasets/sentence-transformers/natural-questions) dataset. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.
This is a test model to experiment with the proposed `DebiasedMultipleNegativesRankingLoss` from [Pull Request #3148](https://github.com/UKPLab/sentence-transformers/pull/3148) in the Sentence Transformers repository, using commit `370bf473e60b57f7d01a6e084b5acaabdac38a2c`.
## Model Details
### Model Description
- **Model Type:** Sentence Transformer
- **Base model:** [answerdotai/ModernBERT-base](https://huggingface.co/answerdotai/ModernBERT-base) <!-- at revision 6e461621ae9e2dffc138de99490e9baee354deb5 -->
- **Maximum Sequence Length:** 8192 tokens
- **Output Dimensionality:** 768 dimensions
- **Similarity Function:** Cosine Similarity
- **Training Dataset:**
- [natural-questions](https://huggingface.co/datasets/sentence-transformers/natural-questions)
- **Language:** en
<!-- - **License:** Unknown -->
### Model Sources
- **Documentation:** [Sentence Transformers Documentation](https://sbert.net)
- **Repository:** [Sentence Transformers on GitHub](https://github.com/UKPLab/sentence-transformers)
- **Hugging Face:** [Sentence Transformers on Hugging Face](https://huggingface.co/models?library=sentence-transformers)
### Full Model Architecture
```
SentenceTransformer(
(0): Transformer({'max_seq_length': 8192, 'do_lower_case': False}) with Transformer model: ModernBertModel
(1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
)
```
## Usage
### Direct Usage (Sentence Transformers)
First install the Sentence Transformers library:
```bash
pip install -U sentence-transformers
```
Then you can load this model and run inference.
```python
from sentence_transformers import SentenceTransformer
# Download from the 🤗 Hub
model = SentenceTransformer("tomaarsen/ModernBERT-base-nq-mnrl")
# Run inference
sentences = [
'what is meaning of am and pm in time',
'12-hour clock The 12-hour clock is a time convention in which the 24 hours of the day are divided into two periods:[1] a.m. (from the Latin, ante meridiem, meaning before midday) and p.m. (post meridiem, meaning past midday).[2] Each period consists of 12 hours numbered: 12 (acting as zero),[3] 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, and 11. The 24 hour/day cycle starts at 12 midnight (often indicated as 12 a.m.), runs through 12 noon (often indicated as 12 p.m.), and continues to the midnight at the end of the day. The 12-hour clock was developed over time from the mid-second millennium BC to the 16th century AD.',
"America's Got Talent America's Got Talent (often abbreviated as AGT) is a televised American talent show competition, broadcast on the NBC television network. It is part of the global Got Talent franchise created by Simon Cowell, and is produced by Fremantle North America and SYCOtv, with distribution done by Fremantle. Since its premiere in June 2006, each season is run during the network's summer schedule, with the show having featured various hosts - it is currently hosted by Tyra Banks, since 2017.[2] It is the first global edition of the franchise, after plans for a British edition in 2005 were suspended, following a dispute between Paul O'Grady, the planned host, and the British broadcaster ITV; production of this edition later resumed in 2007.[3]",
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 768]
# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]
```
<!--
### Direct Usage (Transformers)
<details><summary>Click to see the direct usage in Transformers</summary>
</details>
-->
<!--
### Downstream Usage (Sentence Transformers)
You can finetune this model on your own dataset.
<details><summary>Click to expand</summary>
</details>
-->
<!--
### Out-of-Scope Use
*List how the model may foreseeably be misused and address what users ought not to do with the model.*
-->
## Evaluation
### Metrics
#### Information Retrieval
* Datasets: `NanoMSMARCO` and `NanoHotpotQA`
* Evaluated with [<code>InformationRetrievalEvaluator</code>](https://sbert.net/docs/package_reference/sentence_transformer/evaluation.html#sentence_transformers.evaluation.InformationRetrievalEvaluator)
| Metric | NanoMSMARCO | NanoHotpotQA |
|:--------------------|:------------|:-------------|
| cosine_accuracy@1 | 0.24 | 0.54 |
| cosine_accuracy@3 | 0.44 | 0.62 |
| cosine_accuracy@5 | 0.58 | 0.64 |
| cosine_accuracy@10 | 0.72 | 0.74 |
| cosine_precision@1 | 0.24 | 0.54 |
| cosine_precision@3 | 0.1467 | 0.28 |
| cosine_precision@5 | 0.116 | 0.176 |
| cosine_precision@10 | 0.072 | 0.11 |
| cosine_recall@1 | 0.24 | 0.27 |
| cosine_recall@3 | 0.44 | 0.42 |
| cosine_recall@5 | 0.58 | 0.44 |
| cosine_recall@10 | 0.72 | 0.55 |
| **cosine_ndcg@10** | **0.4603** | **0.4959** |
| cosine_mrr@10 | 0.3797 | 0.5962 |
| cosine_map@100 | 0.3945 | 0.432 |
#### Nano BEIR
* Dataset: `NanoBEIR_mean`
* Evaluated with [<code>NanoBEIREvaluator</code>](https://sbert.net/docs/package_reference/sentence_transformer/evaluation.html#sentence_transformers.evaluation.NanoBEIREvaluator)
| Metric | Value |
|:--------------------|:-----------|
| cosine_accuracy@1 | 0.39 |
| cosine_accuracy@3 | 0.53 |
| cosine_accuracy@5 | 0.61 |
| cosine_accuracy@10 | 0.73 |
| cosine_precision@1 | 0.39 |
| cosine_precision@3 | 0.2133 |
| cosine_precision@5 | 0.146 |
| cosine_precision@10 | 0.091 |
| cosine_recall@1 | 0.255 |
| cosine_recall@3 | 0.43 |
| cosine_recall@5 | 0.51 |
| cosine_recall@10 | 0.635 |
| **cosine_ndcg@10** | **0.4781** |
| cosine_mrr@10 | 0.488 |
| cosine_map@100 | 0.4133 |
<!--
## Bias, Risks and Limitations
*What are the known or foreseeable issues stemming from this model? You could also flag here known failure cases or weaknesses of the model.*
-->
<!--
### Recommendations
*What are recommendations with respect to the foreseeable issues? For example, filtering explicit content.*
-->
## Training Details
### Training Dataset
#### natural-questions
* Dataset: [natural-questions](https://huggingface.co/datasets/sentence-transformers/natural-questions) at [f9e894e](https://huggingface.co/datasets/sentence-transformers/natural-questions/tree/f9e894e1081e206e577b4eaa9ee6de2b06ae6f17)
* Size: 100,231 training samples
* Columns: <code>query</code> and <code>answer</code>
* Approximate statistics based on the first 1000 samples:
| | query | answer |
|:--------|:-----------------------------------------------------------------------------------|:-------------------------------------------------------------------------------------|
| type | string | string |
| details | <ul><li>min: 10 tokens</li><li>mean: 12.46 tokens</li><li>max: 25 tokens</li></ul> | <ul><li>min: 16 tokens</li><li>mean: 139.02 tokens</li><li>max: 537 tokens</li></ul> |
* Samples:
| query | answer |
|:------------------------------------------------------------------------|:---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| <code>who is required to report according to the hmda</code> | <code>Home Mortgage Disclosure Act US financial institutions must report HMDA data to their regulator if they meet certain criteria, such as having assets above a specific threshold. The criteria is different for depository and non-depository institutions and are available on the FFIEC website.[4] In 2012, there were 7,400 institutions that reported a total of 18.7 million HMDA records.[5]</code> |
| <code>what is the definition of endoplasmic reticulum in biology</code> | <code>Endoplasmic reticulum The endoplasmic reticulum (ER) is a type of organelle in eukaryotic cells that forms an interconnected network of flattened, membrane-enclosed sacs or tube-like structures known as cisternae. The membranes of the ER are continuous with the outer nuclear membrane. The endoplasmic reticulum occurs in most types of eukaryotic cells, but is absent from red blood cells and spermatozoa. There are two types of endoplasmic reticulum: rough and smooth. The outer (cytosolic) face of the rough endoplasmic reticulum is studded with ribosomes that are the sites of protein synthesis. The rough endoplasmic reticulum is especially prominent in cells such as hepatocytes. The smooth endoplasmic reticulum lacks ribosomes and functions in lipid manufacture and metabolism, the production of steroid hormones, and detoxification.[1] The smooth ER is especially abundant in mammalian liver and gonad cells. The lacy membranes of the endoplasmic reticulum were first seen in 1945 using elect...</code> |
| <code>what does the ski mean in polish names</code> | <code>Polish name Since the High Middle Ages, Polish-sounding surnames ending with the masculine -ski suffix, including -cki and -dzki, and the corresponding feminine suffix -ska/-cka/-dzka were associated with the nobility (Polish szlachta), which alone, in the early years, had such suffix distinctions.[1] They are widely popular today.</code> |
* Loss: [<code>MultipleNegativesRankingLoss</code>](https://sbert.net/docs/package_reference/sentence_transformer/losses.html#multiplenegativesrankingloss) with these parameters:
```json
{
"scale": 20.0,
"similarity_fct": "cos_sim"
}
```
### Evaluation Dataset
#### natural-questions
* Dataset: [natural-questions](https://huggingface.co/datasets/sentence-transformers/natural-questions) at [f9e894e](https://huggingface.co/datasets/sentence-transformers/natural-questions/tree/f9e894e1081e206e577b4eaa9ee6de2b06ae6f17)
* Size: 100,231 evaluation samples
* Columns: <code>query</code> and <code>answer</code>
* Approximate statistics based on the first 1000 samples:
| | query | answer |
|:--------|:-----------------------------------------------------------------------------------|:------------------------------------------------------------------------------------|
| type | string | string |
| details | <ul><li>min: 10 tokens</li><li>mean: 12.46 tokens</li><li>max: 22 tokens</li></ul> | <ul><li>min: 12 tokens</li><li>mean: 138.0 tokens</li><li>max: 649 tokens</li></ul> |
* Samples:
| query | answer |
|:------------------------------------------------------------------|:--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| <code>difference between russian blue and british blue cat</code> | <code>Russian Blue The coat is known as a "double coat", with the undercoat being soft, downy and equal in length to the guard hairs, which are an even blue with silver tips. However, the tail may have a few very dull, almost unnoticeable stripes. The coat is described as thick, plush and soft to the touch. The feeling is softer than the softest silk. The silver tips give the coat a shimmering appearance. Its eyes are almost always a dark and vivid green. Any white patches of fur or yellow eyes in adulthood are seen as flaws in show cats.[3] Russian Blues should not be confused with British Blues (which are not a distinct breed, but rather a British Shorthair with a blue coat as the British Shorthair breed itself comes in a wide variety of colors and patterns), nor the Chartreux or Korat which are two other naturally occurring breeds of blue cats, although they have similar traits.</code> |
| <code>who played the little girl on mrs doubtfire</code> | <code>Mara Wilson Mara Elizabeth Wilson[2] (born July 24, 1987) is an American writer and former child actress. She is known for playing Natalie Hillard in Mrs. Doubtfire (1993), Susan Walker in Miracle on 34th Street (1994), Matilda Wormwood in Matilda (1996) and Lily Stone in Thomas and the Magic Railroad (2000). Since retiring from film acting, Wilson has focused on writing.</code> |
| <code>what year did the movie the sound of music come out</code> | <code>The Sound of Music (film) The film was released on March 2, 1965 in the United States, initially as a limited roadshow theatrical release. Although critical response to the film was widely mixed, the film was a major commercial success, becoming the number one box office movie after four weeks, and the highest-grossing film of 1965. By November 1966, The Sound of Music had become the highest-grossing film of all-time—surpassing Gone with the Wind—and held that distinction for five years. The film was just as popular throughout the world, breaking previous box-office records in twenty-nine countries. Following an initial theatrical release that lasted four and a half years, and two successful re-releases, the film sold 283 million admissions worldwide and earned a total worldwide gross of $286,000,000.</code> |
* Loss: [<code>MultipleNegativesRankingLoss</code>](https://sbert.net/docs/package_reference/sentence_transformer/losses.html#multiplenegativesrankingloss) with these parameters:
```json
{
"scale": 20.0,
"similarity_fct": "cos_sim"
}
```
### Training Hyperparameters
#### Non-Default Hyperparameters
- `eval_strategy`: steps
- `per_device_train_batch_size`: 128
- `per_device_eval_batch_size`: 128
- `learning_rate`: 8e-05
- `num_train_epochs`: 1
- `warmup_ratio`: 0.05
- `seed`: 12
- `bf16`: True
- `batch_sampler`: no_duplicates
#### All Hyperparameters
<details><summary>Click to expand</summary>
- `overwrite_output_dir`: False
- `do_predict`: False
- `eval_strategy`: steps
- `prediction_loss_only`: True
- `per_device_train_batch_size`: 128
- `per_device_eval_batch_size`: 128
- `per_gpu_train_batch_size`: None
- `per_gpu_eval_batch_size`: None
- `gradient_accumulation_steps`: 1
- `eval_accumulation_steps`: None
- `torch_empty_cache_steps`: None
- `learning_rate`: 8e-05
- `weight_decay`: 0.0
- `adam_beta1`: 0.9
- `adam_beta2`: 0.999
- `adam_epsilon`: 1e-08
- `max_grad_norm`: 1.0
- `num_train_epochs`: 1
- `max_steps`: -1
- `lr_scheduler_type`: linear
- `lr_scheduler_kwargs`: {}
- `warmup_ratio`: 0.05
- `warmup_steps`: 0
- `log_level`: passive
- `log_level_replica`: warning
- `log_on_each_node`: True
- `logging_nan_inf_filter`: True
- `save_safetensors`: True
- `save_on_each_node`: False
- `save_only_model`: False
- `restore_callback_states_from_checkpoint`: False
- `no_cuda`: False
- `use_cpu`: False
- `use_mps_device`: False
- `seed`: 12
- `data_seed`: None
- `jit_mode_eval`: False
- `use_ipex`: False
- `bf16`: True
- `fp16`: False
- `fp16_opt_level`: O1
- `half_precision_backend`: auto
- `bf16_full_eval`: False
- `fp16_full_eval`: False
- `tf32`: None
- `local_rank`: 0
- `ddp_backend`: None
- `tpu_num_cores`: None
- `tpu_metrics_debug`: False
- `debug`: []
- `dataloader_drop_last`: False
- `dataloader_num_workers`: 0
- `dataloader_prefetch_factor`: None
- `past_index`: -1
- `disable_tqdm`: False
- `remove_unused_columns`: True
- `label_names`: None
- `load_best_model_at_end`: False
- `ignore_data_skip`: False
- `fsdp`: []
- `fsdp_min_num_params`: 0
- `fsdp_config`: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
- `fsdp_transformer_layer_cls_to_wrap`: None
- `accelerator_config`: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
- `deepspeed`: None
- `label_smoothing_factor`: 0.0
- `optim`: adamw_torch
- `optim_args`: None
- `adafactor`: False
- `group_by_length`: False
- `length_column_name`: length
- `ddp_find_unused_parameters`: None
- `ddp_bucket_cap_mb`: None
- `ddp_broadcast_buffers`: False
- `dataloader_pin_memory`: True
- `dataloader_persistent_workers`: False
- `skip_memory_metrics`: True
- `use_legacy_prediction_loop`: False
- `push_to_hub`: False
- `resume_from_checkpoint`: None
- `hub_model_id`: None
- `hub_strategy`: every_save
- `hub_private_repo`: None
- `hub_always_push`: False
- `gradient_checkpointing`: False
- `gradient_checkpointing_kwargs`: None
- `include_inputs_for_metrics`: False
- `include_for_metrics`: []
- `eval_do_concat_batches`: True
- `fp16_backend`: auto
- `push_to_hub_model_id`: None
- `push_to_hub_organization`: None
- `mp_parameters`:
- `auto_find_batch_size`: False
- `full_determinism`: False
- `torchdynamo`: None
- `ray_scope`: last
- `ddp_timeout`: 1800
- `torch_compile`: False
- `torch_compile_backend`: None
- `torch_compile_mode`: None
- `dispatch_batches`: None
- `split_batches`: None
- `include_tokens_per_second`: False
- `include_num_input_tokens_seen`: False
- `neftune_noise_alpha`: None
- `optim_target_modules`: None
- `batch_eval_metrics`: False
- `eval_on_start`: False
- `use_liger_kernel`: False
- `eval_use_gather_object`: False
- `average_tokens_across_devices`: False
- `prompts`: None
- `batch_sampler`: no_duplicates
- `multi_dataset_batch_sampler`: proportional
</details>
### Training Logs
| Epoch | Step | Training Loss | Validation Loss | NanoMSMARCO_cosine_ndcg@10 | NanoHotpotQA_cosine_ndcg@10 | NanoBEIR_mean_cosine_ndcg@10 |
|:------:|:----:|:-------------:|:---------------:|:--------------------------:|:---------------------------:|:----------------------------:|
| 0 | 0 | - | - | 0.0785 | 0.1489 | 0.1137 |
| 0.0129 | 10 | 4.1104 | - | - | - | - |
| 0.0258 | 20 | 2.2261 | - | - | - | - |
| 0.0387 | 30 | 1.1152 | - | - | - | - |
| 0.0515 | 40 | 0.504 | - | - | - | - |
| 0.0644 | 50 | 0.2746 | 0.2962 | 0.4216 | 0.4409 | 0.4312 |
| 0.0773 | 60 | 0.2547 | - | - | - | - |
| 0.0902 | 70 | 0.174 | - | - | - | - |
| 0.1031 | 80 | 0.1816 | - | - | - | - |
| 0.1160 | 90 | 0.1554 | - | - | - | - |
| 0.1289 | 100 | 0.1537 | 0.1942 | 0.4321 | 0.4338 | 0.4330 |
| 0.1418 | 110 | 0.1369 | - | - | - | - |
| 0.1546 | 120 | 0.1379 | - | - | - | - |
| 0.1675 | 130 | 0.1388 | - | - | - | - |
| 0.1804 | 140 | 0.1141 | - | - | - | - |
| 0.1933 | 150 | 0.1339 | 0.1541 | 0.4348 | 0.4336 | 0.4342 |
| 0.2062 | 160 | 0.1082 | - | - | - | - |
| 0.2191 | 170 | 0.1115 | - | - | - | - |
| 0.2320 | 180 | 0.1312 | - | - | - | - |
| 0.2448 | 190 | 0.107 | - | - | - | - |
| 0.2577 | 200 | 0.1081 | 0.1281 | 0.4309 | 0.4612 | 0.4460 |
| 0.2706 | 210 | 0.0858 | - | - | - | - |
| 0.2835 | 220 | 0.0903 | - | - | - | - |
| 0.2964 | 230 | 0.0982 | - | - | - | - |
| 0.3093 | 240 | 0.1114 | - | - | - | - |
| 0.3222 | 250 | 0.0766 | 0.1159 | 0.4683 | 0.4655 | 0.4669 |
| 0.3351 | 260 | 0.0842 | - | - | - | - |
| 0.3479 | 270 | 0.0772 | - | - | - | - |
| 0.3608 | 280 | 0.0946 | - | - | - | - |
| 0.3737 | 290 | 0.0818 | - | - | - | - |
| 0.3866 | 300 | 0.0901 | 0.1114 | 0.4376 | 0.4689 | 0.4533 |
| 0.3995 | 310 | 0.0762 | - | - | - | - |
| 0.4124 | 320 | 0.0891 | - | - | - | - |
| 0.4253 | 330 | 0.0786 | - | - | - | - |
| 0.4381 | 340 | 0.0741 | - | - | - | - |
| 0.4510 | 350 | 0.073 | 0.1001 | 0.4586 | 0.4579 | 0.4583 |
| 0.4639 | 360 | 0.074 | - | - | - | - |
| 0.4768 | 370 | 0.0883 | - | - | - | - |
| 0.4897 | 380 | 0.0747 | - | - | - | - |
| 0.5026 | 390 | 0.0637 | - | - | - | - |
| 0.5155 | 400 | 0.0711 | 0.1035 | 0.4331 | 0.4538 | 0.4435 |
| 0.5284 | 410 | 0.0813 | - | - | - | - |
| 0.5412 | 420 | 0.0643 | - | - | - | - |
| 0.5541 | 430 | 0.0793 | - | - | - | - |
| 0.5670 | 440 | 0.0815 | - | - | - | - |
| 0.5799 | 450 | 0.0712 | 0.0953 | 0.4331 | 0.4684 | 0.4507 |
| 0.5928 | 460 | 0.0664 | - | - | - | - |
| 0.6057 | 470 | 0.0637 | - | - | - | - |
| 0.6186 | 480 | 0.0753 | - | - | - | - |
| 0.6314 | 490 | 0.0734 | - | - | - | - |
| 0.6443 | 500 | 0.0755 | 0.0850 | 0.4840 | 0.4443 | 0.4641 |
| 0.6572 | 510 | 0.0676 | - | - | - | - |
| 0.6701 | 520 | 0.071 | - | - | - | - |
| 0.6830 | 530 | 0.0725 | - | - | - | - |
| 0.6959 | 540 | 0.0536 | - | - | - | - |
| 0.7088 | 550 | 0.0532 | 0.0807 | 0.4854 | 0.4601 | 0.4727 |
| 0.7216 | 560 | 0.0601 | - | - | - | - |
| 0.7345 | 570 | 0.0672 | - | - | - | - |
| 0.7474 | 580 | 0.0635 | - | - | - | - |
| 0.7603 | 590 | 0.0691 | - | - | - | - |
| 0.7732 | 600 | 0.0668 | 0.0836 | 0.4690 | 0.4829 | 0.4759 |
| 0.7861 | 610 | 0.0493 | - | - | - | - |
| 0.7990 | 620 | 0.0543 | - | - | - | - |
| 0.8119 | 630 | 0.0574 | - | - | - | - |
| 0.8247 | 640 | 0.0546 | - | - | - | - |
| 0.8376 | 650 | 0.0581 | 0.0834 | 0.4407 | 0.4817 | 0.4612 |
| 0.8505 | 660 | 0.0645 | - | - | - | - |
| 0.8634 | 670 | 0.059 | - | - | - | - |
| 0.8763 | 680 | 0.0604 | - | - | - | - |
| 0.8892 | 690 | 0.0547 | - | - | - | - |
| 0.9021 | 700 | 0.0561 | 0.0796 | 0.4457 | 0.4769 | 0.4613 |
| 0.9149 | 710 | 0.0491 | - | - | - | - |
| 0.9278 | 720 | 0.0505 | - | - | - | - |
| 0.9407 | 730 | 0.0545 | - | - | - | - |
| 0.9536 | 740 | 0.0445 | - | - | - | - |
| 0.9665 | 750 | 0.057 | 0.0765 | 0.4668 | 0.4936 | 0.4802 |
| 0.9794 | 760 | 0.0491 | - | - | - | - |
| 0.9923 | 770 | 0.0526 | - | - | - | - |
| 1.0 | 776 | - | - | 0.4603 | 0.4959 | 0.4781 |
### Framework Versions
- Python: 3.11.10
- Sentence Transformers: 3.4.0.dev0
- Transformers: 4.48.0.dev0
- PyTorch: 2.6.0.dev20241112+cu121
- Accelerate: 1.2.0
- Datasets: 3.2.0
- Tokenizers: 0.21.0
## Citation
### BibTeX
#### Sentence Transformers
```bibtex
@inproceedings{reimers-2019-sentence-bert,
title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
author = "Reimers, Nils and Gurevych, Iryna",
booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
month = "11",
year = "2019",
publisher = "Association for Computational Linguistics",
url = "https://arxiv.org/abs/1908.10084",
}
```
#### MultipleNegativesRankingLoss
```bibtex
@misc{henderson2017efficient,
title={Efficient Natural Language Response Suggestion for Smart Reply},
author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
year={2017},
eprint={1705.00652},
archivePrefix={arXiv},
primaryClass={cs.CL}
}
```
<!--
## Glossary
*Clearly define terms in order to be accessible across audiences.*
-->
<!--
## Model Card Authors
*Lists the people who create the model card, providing recognition and accountability for the detailed work that goes into its construction.*
-->
<!--
## Model Card Contact
*Provides a way for people who have updates to the Model Card, suggestions, or questions, to contact the Model Card authors.*
--> |