File size: 57,170 Bytes
1663fca |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 511 512 513 514 515 516 517 518 519 520 521 522 523 524 525 526 527 528 529 530 531 532 533 534 535 536 537 538 539 540 541 542 543 544 545 546 547 548 549 550 551 552 553 554 555 556 557 558 559 560 561 562 563 564 565 566 567 568 569 570 571 572 573 574 575 576 577 578 579 580 581 582 583 584 585 586 587 588 589 590 591 592 593 594 595 596 597 598 599 600 601 602 603 604 605 606 607 608 609 610 611 612 613 614 615 616 617 618 619 620 621 622 623 624 625 626 627 628 629 630 631 632 633 634 635 636 637 638 639 640 641 642 643 644 645 646 647 648 649 650 651 652 653 654 655 656 657 658 659 660 661 662 663 664 665 666 667 668 669 670 671 672 673 674 675 676 677 678 679 680 681 682 683 684 685 686 687 688 689 690 691 692 693 694 695 696 697 698 699 700 701 702 703 704 705 706 707 708 709 710 711 712 713 714 715 716 717 718 719 720 721 722 723 724 725 726 727 728 729 730 731 732 733 734 735 736 737 738 739 740 741 742 743 744 745 746 747 748 749 750 |
---
base_model: BAAI/bge-small-en
datasets:
- sentence-transformers/hotpotqa
language:
- en
library_name: sentence-transformers
license: apache-2.0
metrics:
- cosine_accuracy
- dot_accuracy
- manhattan_accuracy
- euclidean_accuracy
- max_accuracy
pipeline_tag: sentence-similarity
tags:
- sentence-transformers
- sentence-similarity
- feature-extraction
- generated_from_trainer
- dataset_size:76064
- loss:MatryoshkaLoss
- loss:TripletLoss
widget:
- source_sentence: When was the brewery founded by the family who resided in the Lemp
Mansion founded?
sentences:
- Latrobe Brewing Company Latrobe Brewing Company in Latrobe, Pennsylvania, founded
in 1839, was one of the largest breweries in the United States and the maker of
"Rolling Rock" beer (famous for its small green bottles). It was purchased by
Labatt Brewing Company in 1987, which in turn was purchased in 1995 by the Belgian
brewing conglomerate corporation Interbrew, which merged later into InBev in 2004.
- Lemp Mansion The Lemp Mansion (3322 DeMenil Place, St. Louis, Missouri) is a historical
house in Benton Park, St. Louis, Missouri. It is also the site of four suicides
by Lemp family members after the death of the son Frederick Lemp, whose William
J. Lemp Brewing Co. dominated the St. Louis beer market before Prohibition with
its Falstaff beer brand. The mansion is said to be haunted by members of the Lemp
family.
- 'Man of the House (song) "Man of the House" is a song co-written and recorded
by American country music artist Chuck Wicks. It was released in January 2009
as the third single from the album "Starting Now". The song reached #27 on the
"Billboard" Hot Country Songs chart. The song was written by Wicks and Michael
Mobley.'
- source_sentence: Who is the translator and researcher who served as a dean at Hartford
Junior College and translated her Belgian-French partner's 1968 work into English?
sentences:
- William L. Shirer William Lawrence Shirer (February 23, 1904 – December 28, 1993)
was an American journalist and war correspondent. He wrote "The Rise and Fall
of the Third Reich", a history of Nazi Germany that has been read by many and
cited in scholarly works for more than 50 years. Originally a foreign correspondent
for the "Chicago Tribune" and the International News Service, Shirer was the first
reporter hired by Edward R. Murrow for what would become a CBS radio team of journalists
known as "Murrow's Boys". He became known for his broadcasts from Berlin, from
the rise of the Nazi dictatorship through the first year of World War II (1940).
With Murrow, he organized the first broadcast world news roundup, a format still
followed by news broadcasts.
- 'The Abyss (Yourcenar novel) The Abyss (French: L''Œuvre au noir ) is a 1968 novel
by the Belgian-French writer Marguerite Yourcenar. Its narrative centers on the
life and death of Zeno, a physician, philosopher, scientist and alchemist born
in Bruges during the Renaissance era. The book was published in France in 1968
and was met with immediate popular interest as well as critical acclaim, obtaining
the Prix Femina with unanimous votes the year of its publication. The English
translation by Grace Frick has been published under the title "The Abyss" or alternatively
Zeno of Bruges. Belgian filmmaker André Delvaux adapted it into a film in 1988.'
- Pierre L. van den Berghe Pierre L. van den Berghe (born 1933) is professor emeritus
of sociology and anthropology at the University of Washington, where he has worked
since 1965. Born in the Belgian Congo to Belgian parents, and spending World War
II in occupied Belgium, he was an early witness to ethnic conflict and racism,
which eventually led him to become a leading authority on ethnic relations. He
has conducted field work in South Africa, Mexico, Guatemala, Iran, Lebanon, Nigeria,
Peru, and Israel. Early in his career, he lectured at the University of Natal
alongside Leo Kuper and Fatima Meer. A student of Talcott Parsons at Harvard (receiving
the Ph.D. in 1960), he nevertheless had little interest in structural functionalism
and was one of the first proponents of sociobiological approaches to social phenomena.
- source_sentence: Them Crooked Vultures and The Vines are both part of what music
genres?
sentences:
- Friendly Center Friendly Center is a large, open-air shopping center located in
northwestern Greensboro, North Carolina, near the intersection of Wendover Avenue
and Friendly Avenue. The shopping center opened in August 1957, and with its inward
orientation, Friendly Center could be classified as an outdoor lifestyle center.
Its anchor tenants include Belk, Macy's, and Sears. Other tenants include Barnes
& Noble, Old Navy, The Grande Theatre is a 16-screen multiplex cinema operated
by Regal Cinemas. It also contains Harris Teeter's flagship supermarket location
encompassing 72,000 square feet (6,700 m2) and Whole Foods Market. There are specialty
"foodie" stores tucked away in the back corner by Harris Teeter such as the Savory
Spice Shop and Midtown Olive Oil. It features a number of national retailers such
as Banana Republic, Victoria's Secret, The Limited, Bath & Body Works, Express,
The GAP, Eddie Bauer, Talbots, Birkenstock Feet First, Pier 1, and New York &
Company.
- Twin Wild Twin Wild is a British four-piece alternative rock band. Formed in 2012,
the band is made up of the collective creative energies of Richard Hutchison (vocals,
guitars), Imran Mair (drums), David Cuzner (guitars) and Edward Thomas (bass).
In 2014, the band self-released their track "Fears", which garnered over half
a million plays on Soundcloud and charted in Spotify’s Top 20 viral chart. Their
style of music has been compared to the likes of Foals, The Neighbourhood, and
Bastille. The band have been hailed by Edith Bowman as "The love child of Bastille
and Biffy Clyro".
- Them Crooked Vultures Them Crooked Vultures is a rock supergroup formed in Los
Angeles in 2009 by John Paul Jones (former member of Led Zeppelin) on bass and
keyboards, Dave Grohl (of Foo Fighters and formerly of Nirvana) on drums and backing
vocals, and Josh Homme (of Queens of the Stone Age, Eagles of Death Metal and
formerly of Kyuss) on guitar and vocals. The group also includes guitarist Alain
Johannes during live performances. The band began recording in February 2009,
and performed their first gig on August 9, 2009, in Chicago, followed by a European
debut on August 19. On October 1 the group embarked on a worldwide tour titled
"Deserve the Future" with dates going into 2010. The band's first single "New
Fang" was released in October 2009, followed by the group's self-titled debut
album the following month, debuting at number 12 on the "Billboard" 200. The group
won the 2011 Grammy Award for Best Hard Rock Performance for "New Fang".
- source_sentence: What type of collection does Nådens år and Agnetha Fältskog have
in common?
sentences:
- Gardiner Island (Nunavut) Gardiner Island is one of the many uninhabited Canadian
arctic islands in Qikiqtaaluk Region, Nunavut. It is a Baffin Island offshore
island located in Frobisher Bay south of the capital city of Iqaluit.
- Nådens år Nådens år (The Year of Grace) is the third studio album by the Swedish
rock artist Ulf Lundell. It was released in April 1978 on EMI and Parlophone.
It was recorded in EMI Studio, Stockholm, and produced by Kjell Andersson and
Lundell. It includes "Snön faller och vi med den" ("The snow is falling and we
are too"), one of Lundell's more famous songs. Agnetha Fältskog is involved in
the song. The cover picture shows Lundell sitting on a rock next to a dog and
was taken in Åre in 1977. "Nådens år" achieved Gold status in Sweden.
- Åsa Elzén Åsa Elzén is an artist whose work is informed by feminist theory, intersectionality
and post-colonialism. Elzén was born in Sweden in 1972 and currently lives and
works in Berlin.
- source_sentence: Red Velvet is a 2012 play by Lolita Chakrabarti, dealing with the
biography of a 19th century actor born in which year ?
sentences:
- Herbert Campbell Herbert Campbell (22 December 1844 – 19 July 1904) born Herbert
Edward Story was an English comedian and actor who appeared in music hall, Victorian
burlesques and musical comedies during the Victorian era. He was famous for starring,
for forty years, in the Theatre Royal, Drury Lane's annual Christmas pantomimes,
predominantly as a dame.
- Leptinella Leptinella is a genus of alpine flowering plant in the Asteraceae family,
comprising 33 species, distributed in New Guinea, Australia, New Zealand, South
Africa, and South America. Many of the species are endemic to New Zealand.
- Red Velvet (play) Red Velvet is a 2012 play by Lolita Chakrabarti, dealing with
the biography of the 19th century actor Ira Aldridge and his taking the role of
"Othello". It premiered at the Tricycle Theatre (directed by its new artistic
director Indhu Rubasingham) from 11 October to 24 November 2012, with Aldridge
played by Adrian Lester.
model-index:
- name: BGE-base-en-v1.5-Hotpotqa
results:
- task:
type: triplet
name: Triplet
dataset:
name: dim 384
type: dim_384
metrics:
- type: cosine_accuracy
value: 0.8698532891623284
name: Cosine Accuracy
- type: dot_accuracy
value: 0.13014671083767157
name: Dot Accuracy
- type: manhattan_accuracy
value: 0.8737576904874585
name: Manhattan Accuracy
- type: euclidean_accuracy
value: 0.8698532891623284
name: Euclidean Accuracy
- type: max_accuracy
value: 0.8737576904874585
name: Max Accuracy
- task:
type: triplet
name: Triplet
dataset:
name: dim 256
type: dim_256
metrics:
- type: cosine_accuracy
value: 0.86819687647894
name: Cosine Accuracy
- type: dot_accuracy
value: 0.1319214387127307
name: Dot Accuracy
- type: manhattan_accuracy
value: 0.8762423095125415
name: Manhattan Accuracy
- type: euclidean_accuracy
value: 0.8680785612872692
name: Euclidean Accuracy
- type: max_accuracy
value: 0.8762423095125415
name: Max Accuracy
- task:
type: triplet
name: Triplet
dataset:
name: dim 128
type: dim_128
metrics:
- type: cosine_accuracy
value: 0.8663038334122102
name: Cosine Accuracy
- type: dot_accuracy
value: 0.1386654046379555
name: Dot Accuracy
- type: manhattan_accuracy
value: 0.8742309512541411
name: Manhattan Accuracy
- type: euclidean_accuracy
value: 0.865712257453857
name: Euclidean Accuracy
- type: max_accuracy
value: 0.8742309512541411
name: Max Accuracy
- task:
type: triplet
name: Triplet
dataset:
name: dim 64
type: dim_64
metrics:
- type: cosine_accuracy
value: 0.8634642688121155
name: Cosine Accuracy
- type: dot_accuracy
value: 0.1508518693800284
name: Dot Accuracy
- type: manhattan_accuracy
value: 0.8728111689540937
name: Manhattan Accuracy
- type: euclidean_accuracy
value: 0.8637008991954567
name: Euclidean Accuracy
- type: max_accuracy
value: 0.8728111689540937
name: Max Accuracy
---
# BGE-base-en-v1.5-Hotpotqa
This is a [sentence-transformers](https://www.SBERT.net) model finetuned from [BAAI/bge-small-en](https://huggingface.co/BAAI/bge-small-en) on the [sentence-transformers/hotpotqa](https://huggingface.co/datasets/sentence-transformers/hotpotqa) dataset. It maps sentences & paragraphs to a 384-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.
## Model Details
### Model Description
- **Model Type:** Sentence Transformer
- **Base model:** [BAAI/bge-small-en](https://huggingface.co/BAAI/bge-small-en) <!-- at revision 2275a7bdee235e9b4f01fa73aa60d3311983cfea -->
- **Maximum Sequence Length:** 512 tokens
- **Output Dimensionality:** 384 tokens
- **Similarity Function:** Cosine Similarity
- **Training Dataset:**
- [sentence-transformers/hotpotqa](https://huggingface.co/datasets/sentence-transformers/hotpotqa)
- **Language:** en
- **License:** apache-2.0
### Model Sources
- **Documentation:** [Sentence Transformers Documentation](https://sbert.net)
- **Repository:** [Sentence Transformers on GitHub](https://github.com/UKPLab/sentence-transformers)
- **Hugging Face:** [Sentence Transformers on Hugging Face](https://huggingface.co/models?library=sentence-transformers)
### Full Model Architecture
```
SentenceTransformer(
(0): Transformer({'max_seq_length': 512, 'do_lower_case': True}) with Transformer model: BertModel
(1): Pooling({'word_embedding_dimension': 384, 'pooling_mode_cls_token': True, 'pooling_mode_mean_tokens': False, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
(2): Normalize()
)
```
## Usage
### Direct Usage (Sentence Transformers)
First install the Sentence Transformers library:
```bash
pip install -U sentence-transformers
```
Then you can load this model and run inference.
```python
from sentence_transformers import SentenceTransformer
# Download from the 🤗 Hub
model = SentenceTransformer("sentence_transformers_model_id")
# Run inference
sentences = [
'Red Velvet is a 2012 play by Lolita Chakrabarti, dealing with the biography of a 19th century actor born in which year ?',
'Red Velvet (play) Red Velvet is a 2012 play by Lolita Chakrabarti, dealing with the biography of the 19th century actor Ira Aldridge and his taking the role of "Othello". It premiered at the Tricycle Theatre (directed by its new artistic director Indhu Rubasingham) from 11 October to 24 November 2012, with Aldridge played by Adrian Lester.',
"Herbert Campbell Herbert Campbell (22 December 1844 – 19 July 1904) born Herbert Edward Story was an English comedian and actor who appeared in music hall, Victorian burlesques and musical comedies during the Victorian era. He was famous for starring, for forty years, in the Theatre Royal, Drury Lane's annual Christmas pantomimes, predominantly as a dame.",
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 384]
# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]
```
<!--
### Direct Usage (Transformers)
<details><summary>Click to see the direct usage in Transformers</summary>
</details>
-->
<!--
### Downstream Usage (Sentence Transformers)
You can finetune this model on your own dataset.
<details><summary>Click to expand</summary>
</details>
-->
<!--
### Out-of-Scope Use
*List how the model may foreseeably be misused and address what users ought not to do with the model.*
-->
## Evaluation
### Metrics
#### Triplet
* Dataset: `dim_384`
* Evaluated with [<code>TripletEvaluator</code>](https://sbert.net/docs/package_reference/sentence_transformer/evaluation.html#sentence_transformers.evaluation.TripletEvaluator)
| Metric | Value |
|:--------------------|:-----------|
| **cosine_accuracy** | **0.8699** |
| dot_accuracy | 0.1301 |
| manhattan_accuracy | 0.8738 |
| euclidean_accuracy | 0.8699 |
| max_accuracy | 0.8738 |
#### Triplet
* Dataset: `dim_256`
* Evaluated with [<code>TripletEvaluator</code>](https://sbert.net/docs/package_reference/sentence_transformer/evaluation.html#sentence_transformers.evaluation.TripletEvaluator)
| Metric | Value |
|:--------------------|:-----------|
| **cosine_accuracy** | **0.8682** |
| dot_accuracy | 0.1319 |
| manhattan_accuracy | 0.8762 |
| euclidean_accuracy | 0.8681 |
| max_accuracy | 0.8762 |
#### Triplet
* Dataset: `dim_128`
* Evaluated with [<code>TripletEvaluator</code>](https://sbert.net/docs/package_reference/sentence_transformer/evaluation.html#sentence_transformers.evaluation.TripletEvaluator)
| Metric | Value |
|:--------------------|:-----------|
| **cosine_accuracy** | **0.8663** |
| dot_accuracy | 0.1387 |
| manhattan_accuracy | 0.8742 |
| euclidean_accuracy | 0.8657 |
| max_accuracy | 0.8742 |
#### Triplet
* Dataset: `dim_64`
* Evaluated with [<code>TripletEvaluator</code>](https://sbert.net/docs/package_reference/sentence_transformer/evaluation.html#sentence_transformers.evaluation.TripletEvaluator)
| Metric | Value |
|:--------------------|:-----------|
| **cosine_accuracy** | **0.8635** |
| dot_accuracy | 0.1509 |
| manhattan_accuracy | 0.8728 |
| euclidean_accuracy | 0.8637 |
| max_accuracy | 0.8728 |
<!--
## Bias, Risks and Limitations
*What are the known or foreseeable issues stemming from this model? You could also flag here known failure cases or weaknesses of the model.*
-->
<!--
### Recommendations
*What are recommendations with respect to the foreseeable issues? For example, filtering explicit content.*
-->
## Training Details
### Training Dataset
#### sentence-transformers/hotpotqa
* Dataset: [sentence-transformers/hotpotqa](https://huggingface.co/datasets/sentence-transformers/hotpotqa) at [f07d3cd](https://huggingface.co/datasets/sentence-transformers/hotpotqa/tree/f07d3cd2d290ea2e83ed35e33d67d6a4658b8786)
* Size: 76,064 training samples
* Columns: <code>anchor</code>, <code>positive</code>, and <code>negative</code>
* Approximate statistics based on the first 1000 samples:
| | anchor | positive | negative |
|:--------|:----------------------------------------------------------------------------------|:-------------------------------------------------------------------------------------|:------------------------------------------------------------------------------------|
| type | string | string | string |
| details | <ul><li>min: 8 tokens</li><li>mean: 24.08 tokens</li><li>max: 95 tokens</li></ul> | <ul><li>min: 23 tokens</li><li>mean: 100.12 tokens</li><li>max: 512 tokens</li></ul> | <ul><li>min: 15 tokens</li><li>mean: 88.02 tokens</li><li>max: 512 tokens</li></ul> |
* Samples:
| anchor | positive | negative |
|:-------------------------------------------------------------------------------------------------|:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| <code>What nationality was the player named MVP in 2017 World Baseball Classic – Pool C ?</code> | <code>2017 World Baseball Classic – Pool C Pool C of the First Round of the 2017 World Baseball Classic was held at Marlins Park, Miami, Florida, United States, from March 9 to 12, 2017, between Canada, Colombia, the Dominican Republic, and the United States. Pool C was a round-robin tournament. Each team played the other three teams once, with the top two teams – the Dominican Republic and the United States – advancing to Pool F, one of two second-round pools. Manny Machado of the Dominican Republic was named MVP for the first-round Pool C bracket of the WBC, after batting .357.</code> | <code>2017 World Baseball Classic – Qualifier 2 Qualifier 2 of the Qualifying Round of the 2017 World Baseball Classic was held at Estadio B'Air, Mexicali, Mexico from March 17 to 20, 2016.</code> |
| <code>Karl Kraepelin specialized in the study of what predatory arachnids?</code> | <code>Karl Kraepelin Karl Matthias Friedrich Magnus Kraepelin (14 December 1848 Neustrelitz – 28 June 1915 Hamburg), was a German naturalist who specialised in the study of scorpions, centipedes, spiders and solfugids, and was noted for his monograph ""Scorpiones und Pedipalpi"" (Berlin) in 1899, which was an exhaustive survey of the taxonomy of the Order Scorpiones. From 1889–1914 he was Director of the "Naturhistorisches Museum Hamburg ", which was destroyed during World War II, and worked on myriapods from 1901–1916.</code> | <code>Teuthology Teuthology (from Greek "τεῦθος" , "cuttlefish, squid", and -λογία , "-logia") is the study of cephalopods.</code> |
| <code>Who directed the 1990 American crime film in which Vito Pick me played a bit part?</code> | <code>Vito Picone Vito Picone (born March 20, 1941) is the lead singer of The Elegants, and along with Jimmy Mochella is a remaining original member. He has also played bit parts in "Goodfellas", "Analyze This", and "The Sopranos".</code> | <code>The Rookie (1990 film) The Rookie is a 1990 American buddy cop film directed by Clint Eastwood and produced by Howard G. Kazanjian, Steven Siebert and David Valdes. It was written from a screenplay conceived by Boaz Yakin and Scott Spiegel. The film stars Charlie Sheen, Clint Eastwood, Raúl Juliá, Sônia Braga, Lara Flynn Boyle, and Tom Skerritt. Eastwood plays a veteran police officer teamed up with a younger detective played by Sheen ("the rookie"), whose intent is to take down a German crime lord in downtown Los Angeles following months of investigation into an exotic car theft ring.</code> |
* Loss: [<code>MatryoshkaLoss</code>](https://sbert.net/docs/package_reference/sentence_transformer/losses.html#matryoshkaloss) with these parameters:
```json
{
"loss": "TripletLoss",
"matryoshka_dims": [
384,
256,
128,
64
],
"matryoshka_weights": [
1,
1,
1,
1
],
"n_dims_per_step": -1
}
```
### Evaluation Dataset
#### sentence-transformers/hotpotqa
* Dataset: [sentence-transformers/hotpotqa](https://huggingface.co/datasets/sentence-transformers/hotpotqa) at [f07d3cd](https://huggingface.co/datasets/sentence-transformers/hotpotqa/tree/f07d3cd2d290ea2e83ed35e33d67d6a4658b8786)
* Size: 8,452 evaluation samples
* Columns: <code>anchor</code>, <code>positive</code>, and <code>negative</code>
* Approximate statistics based on the first 1000 samples:
| | anchor | positive | negative |
|:--------|:------------------------------------------------------------------------------------|:-------------------------------------------------------------------------------------|:------------------------------------------------------------------------------------|
| type | string | string | string |
| details | <ul><li>min: 10 tokens</li><li>mean: 24.53 tokens</li><li>max: 114 tokens</li></ul> | <ul><li>min: 19 tokens</li><li>mean: 103.87 tokens</li><li>max: 407 tokens</li></ul> | <ul><li>min: 17 tokens</li><li>mean: 88.32 tokens</li><li>max: 356 tokens</li></ul> |
* Samples:
| anchor | positive | negative |
|:---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| <code>Which actress, known for her role as Harper Munroe on the MTV comedy series "Happyland", starred alongside Laura Marano, Parker Mack, Michelle Clunie and Kathleen Wilhoite in the film A Sort of Homecoming?</code> | <code>A Sort of Homecoming (film) A Sort of Homecoming is an American drama directed by Maria Burton, her fifth feature film. The films stars Katherine McNamara, Laura Marano, Parker Mack, Michelle Clunie and Kathleen Wilhoite. The film premiered March 14, 2015 at the Omaha Film Festival.</code> | <code>Nellie Bellflower Nellie Bellflower (born May 1, 1946 in Phoenix, Arizona) is an American actress and voice artist who provided the voice of Princess Ariel in the Ruby-Spears animated television series "Thundarr the Barbarian". She has also been in "The Last Unicorn" (voice), Rankin/Bass "The Return of the King", "Americathon", the miniseries "East of Eden", and guest roles on various TV shows such as "Barnaby Jones", "Barney Miller", "Starsky and Hutch", and "Happy Days" as Fonzie's ex-fiancée Maureen Johnson, a.k.a. "The Lone Stripper", in the Season 2 episode of the series titled "Fonzie's Getting Married" (episode #13). Nellie has been involved in movie production with three projects: "The Girl in Melanie Klein" (2008), "Miss Pettigrew Lives for a Day" (2008) and "Finding Neverland" (2004), for which she was nominated for an Academy Award as Producer for Best Picture. She is married to Michael Mislove.</code> |
| <code>Between Pizza Fusion and Pizzeria Venti, which restaurant emphasizes organic ingredients and green building methods?</code> | <code>Pizza Fusion Pizza Fusion is a Deerfield Beach, Florida-based pizza restaurant chain. Using mostly organic ingredients and emphasizing green building methods, the restaurants operate under the tagline Saving the Earth, One Pizza at a Time.</code> | <code>Pizza Schmizza Pizza Schmizza is an American pizza chain with 23 locations throughout the Portland, Oregon area, and two in southern Oregon. Pizza Schmizza, primarily selling thin crust pizza by-the-slice.</code> |
| <code>What company distributed the stop motion spin-off special "The Year Without a Santa Claus," which aired on December 10, 1974?</code> | <code>A Miser Brothers' Christmas A Miser Brothers' Christmas is a stop motion spin-off special based on some of the characters from the 1974 Rankin-Bass special "The Year Without a Santa Claus". Distributed by Warner Bros. Animation under their Warner Premiere label (the rights holders of the post-1974 Rankin-Bass library) and Toronto-based Cuppa Coffee Studios, the one-hour special premiered on ABC Family on Saturday, December 13, 2008, during the network's annual The 25 Days of Christmas programming. Mickey Rooney and George S. Irving reprised their respective roles as Santa Claus and Heat Miser at ages 88 and 86. Snow Miser, originally portrayed by Dick Shawn who died in 1987, was voiced by Juan Chioran, while Mrs. Claus, voiced by Shirley Booth in the original, was portrayed by Catherine Disher (because Booth had died in 1992). The movie aimed to emulate the Rankin/Bass animation style. This is the last Christmas special to feature Mickey Rooney as Santa Claus, as he died in 2014, as well as the last time George Irving voiced Heat Miser, as he died in 2016.</code> | <code>Holidaze: The Christmas That Almost Didn't Happen Holidaze: The Christmas That Almost Didn't Happen is an American stop-motion animated Christmas television special directed by David H. Brooks, that originally aired in 2006 and produced by BixPix Entertainment, Once Upon a Frog and Madison Street Entertainment. The show's plot has Rusty Reindeer (Fred Savage) the brother of Rudolph the Red Nosed Reindeer joining a support group for depressed holiday icons, and he and the other characters search for the meaning of Christmas and help a young boy (Dylan and Cole Sprouse) to get on Santa's nice list. Rusty's cohorts include Candie, the Easter Bunny (Gladys Knight); Mr. C, the grouchy cherub (Paul Rodriguez); Albert, the Thanksgiving Turkey (Harland Williams); And Trick and Treat (Brenda Song and Emily Osment) the teenage Halloween Ghosts.</code> |
* Loss: [<code>MatryoshkaLoss</code>](https://sbert.net/docs/package_reference/sentence_transformer/losses.html#matryoshkaloss) with these parameters:
```json
{
"loss": "TripletLoss",
"matryoshka_dims": [
384,
256,
128,
64
],
"matryoshka_weights": [
1,
1,
1,
1
],
"n_dims_per_step": -1
}
```
### Training Hyperparameters
#### Non-Default Hyperparameters
- `eval_strategy`: steps
- `per_device_train_batch_size`: 32
- `per_device_eval_batch_size`: 32
- `gradient_accumulation_steps`: 16
- `learning_rate`: 2e-05
- `num_train_epochs`: 50
- `lr_scheduler_type`: cosine
- `warmup_ratio`: 0.1
- `bf16`: True
- `tf32`: True
- `load_best_model_at_end`: True
- `optim`: adamw_torch_fused
- `resume_from_checkpoint`: bge-small-hotpotwa-matryoshka
- `batch_sampler`: no_duplicates
#### All Hyperparameters
<details><summary>Click to expand</summary>
- `overwrite_output_dir`: False
- `do_predict`: False
- `eval_strategy`: steps
- `prediction_loss_only`: True
- `per_device_train_batch_size`: 32
- `per_device_eval_batch_size`: 32
- `per_gpu_train_batch_size`: None
- `per_gpu_eval_batch_size`: None
- `gradient_accumulation_steps`: 16
- `eval_accumulation_steps`: None
- `learning_rate`: 2e-05
- `weight_decay`: 0.0
- `adam_beta1`: 0.9
- `adam_beta2`: 0.999
- `adam_epsilon`: 1e-08
- `max_grad_norm`: 1.0
- `num_train_epochs`: 50
- `max_steps`: -1
- `lr_scheduler_type`: cosine
- `lr_scheduler_kwargs`: {}
- `warmup_ratio`: 0.1
- `warmup_steps`: 0
- `log_level`: passive
- `log_level_replica`: warning
- `log_on_each_node`: True
- `logging_nan_inf_filter`: True
- `save_safetensors`: True
- `save_on_each_node`: False
- `save_only_model`: False
- `restore_callback_states_from_checkpoint`: False
- `no_cuda`: False
- `use_cpu`: False
- `use_mps_device`: False
- `seed`: 42
- `data_seed`: None
- `jit_mode_eval`: False
- `use_ipex`: False
- `bf16`: True
- `fp16`: False
- `fp16_opt_level`: O1
- `half_precision_backend`: auto
- `bf16_full_eval`: False
- `fp16_full_eval`: False
- `tf32`: True
- `local_rank`: 0
- `ddp_backend`: None
- `tpu_num_cores`: None
- `tpu_metrics_debug`: False
- `debug`: []
- `dataloader_drop_last`: False
- `dataloader_num_workers`: 0
- `dataloader_prefetch_factor`: None
- `past_index`: -1
- `disable_tqdm`: False
- `remove_unused_columns`: True
- `label_names`: None
- `load_best_model_at_end`: True
- `ignore_data_skip`: False
- `fsdp`: []
- `fsdp_min_num_params`: 0
- `fsdp_config`: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
- `fsdp_transformer_layer_cls_to_wrap`: None
- `accelerator_config`: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
- `deepspeed`: None
- `label_smoothing_factor`: 0.0
- `optim`: adamw_torch_fused
- `optim_args`: None
- `adafactor`: False
- `group_by_length`: False
- `length_column_name`: length
- `ddp_find_unused_parameters`: None
- `ddp_bucket_cap_mb`: None
- `ddp_broadcast_buffers`: False
- `dataloader_pin_memory`: True
- `dataloader_persistent_workers`: False
- `skip_memory_metrics`: True
- `use_legacy_prediction_loop`: False
- `push_to_hub`: False
- `resume_from_checkpoint`: bge-small-hotpotwa-matryoshka
- `hub_model_id`: None
- `hub_strategy`: every_save
- `hub_private_repo`: False
- `hub_always_push`: False
- `gradient_checkpointing`: False
- `gradient_checkpointing_kwargs`: None
- `include_inputs_for_metrics`: False
- `eval_do_concat_batches`: True
- `fp16_backend`: auto
- `push_to_hub_model_id`: None
- `push_to_hub_organization`: None
- `mp_parameters`:
- `auto_find_batch_size`: False
- `full_determinism`: False
- `torchdynamo`: None
- `ray_scope`: last
- `ddp_timeout`: 1800
- `torch_compile`: False
- `torch_compile_backend`: None
- `torch_compile_mode`: None
- `dispatch_batches`: None
- `split_batches`: None
- `include_tokens_per_second`: False
- `include_num_input_tokens_seen`: False
- `neftune_noise_alpha`: None
- `optim_target_modules`: None
- `batch_eval_metrics`: False
- `batch_sampler`: no_duplicates
- `multi_dataset_batch_sampler`: proportional
</details>
### Training Logs
| Epoch | Step | Training Loss | loss | dim_128_cosine_accuracy | dim_256_cosine_accuracy | dim_384_cosine_accuracy | dim_64_cosine_accuracy |
|:-------:|:----:|:-------------:|:-------:|:-----------------------:|:-----------------------:|:-----------------------:|:----------------------:|
| 0.3366 | 50 | 19.5758 | 19.3933 | 0.9552 | 0.9663 | 0.9668 | 0.9359 |
| 0.6731 | 100 | 19.4573 | 19.0971 | 0.9571 | 0.9646 | 0.9653 | 0.9450 |
| 1.0097 | 150 | 19.1409 | 18.4070 | 0.9385 | 0.9434 | 0.9473 | 0.9307 |
| 1.3462 | 200 | 18.6431 | 17.3292 | 0.9126 | 0.9164 | 0.9184 | 0.9094 |
| 1.6828 | 250 | 18.2288 | 16.8751 | 0.9063 | 0.9071 | 0.9100 | 0.9023 |
| 2.0194 | 300 | 18.0425 | 16.6981 | 0.9020 | 0.9032 | 0.9045 | 0.8990 |
| 2.3559 | 350 | 17.9458 | 16.6155 | 0.9037 | 0.9013 | 0.9022 | 0.8984 |
| 2.6925 | 400 | 17.8525 | 16.5536 | 0.8978 | 0.8971 | 0.8974 | 0.8948 |
| 3.0290 | 450 | 17.7529 | 16.5136 | 0.8980 | 0.8956 | 0.8953 | 0.8951 |
| 3.3656 | 500 | 17.6709 | 16.4824 | 0.8932 | 0.8914 | 0.8928 | 0.8907 |
| 3.7021 | 550 | 17.5348 | 16.4632 | 0.8863 | 0.8858 | 0.8859 | 0.8849 |
| 4.0387 | 600 | 17.4198 | 16.4601 | 0.8852 | 0.8862 | 0.8859 | 0.8839 |
| 4.3753 | 650 | 17.3673 | 16.4405 | 0.8854 | 0.8864 | 0.8865 | 0.8842 |
| 4.7118 | 700 | 17.2603 | 16.4356 | 0.8835 | 0.8838 | 0.8838 | 0.8807 |
| 5.0484 | 750 | 17.1807 | 16.4443 | 0.8850 | 0.8864 | 0.8859 | 0.8838 |
| 5.3849 | 800 | 17.1629 | 16.4202 | 0.8848 | 0.8862 | 0.8867 | 0.8842 |
| 5.7215 | 850 | 17.0747 | 16.4162 | 0.8854 | 0.8875 | 0.8869 | 0.8837 |
| 6.0581 | 900 | 17.0161 | 16.4192 | 0.8852 | 0.8863 | 0.8856 | 0.8856 |
| 6.3946 | 950 | 17.0146 | 16.4033 | 0.8849 | 0.8854 | 0.8856 | 0.8844 |
| 6.7312 | 1000 | 16.9393 | 16.4053 | 0.8829 | 0.8839 | 0.8848 | 0.8835 |
| 7.0677 | 1050 | 16.899 | 16.4162 | 0.8826 | 0.8829 | 0.8833 | 0.8818 |
| 7.4043 | 1100 | 16.9112 | 16.4051 | 0.8829 | 0.8835 | 0.8833 | 0.8820 |
| 7.7408 | 1150 | 16.8508 | 16.4044 | 0.8822 | 0.8825 | 0.8830 | 0.8820 |
| 8.0774 | 1200 | 16.8104 | 16.4063 | 0.8816 | 0.8816 | 0.8814 | 0.8817 |
| 8.4140 | 1250 | 16.8212 | 16.4040 | 0.8835 | 0.8822 | 0.8822 | 0.8820 |
| 8.7505 | 1300 | 16.7743 | 16.3934 | 0.8822 | 0.8824 | 0.8817 | 0.8810 |
| 9.0871 | 1350 | 16.7383 | 16.3963 | 0.8810 | 0.8820 | 0.8807 | 0.8800 |
| 9.4236 | 1400 | 16.743 | 16.4067 | 0.8819 | 0.8822 | 0.8819 | 0.8798 |
| 9.7602 | 1450 | 16.7047 | 16.3959 | 0.8804 | 0.8810 | 0.8810 | 0.8797 |
| 10.0968 | 1500 | 16.6782 | 16.3986 | 0.8788 | 0.8791 | 0.8796 | 0.8784 |
| 10.4333 | 1550 | 16.6708 | 16.4016 | 0.8794 | 0.8792 | 0.8797 | 0.8791 |
| 10.7699 | 1600 | 16.6485 | 16.3963 | 0.8790 | 0.8801 | 0.8791 | 0.8781 |
| 11.1064 | 1650 | 16.6205 | 16.4012 | 0.8779 | 0.8787 | 0.8793 | 0.8771 |
| 11.4430 | 1700 | 16.6095 | 16.4131 | 0.8786 | 0.8790 | 0.8794 | 0.8791 |
| 11.7796 | 1750 | 16.5891 | 16.4070 | 0.8807 | 0.8805 | 0.8810 | 0.8801 |
| 12.1161 | 1800 | 16.5619 | 16.3963 | 0.8794 | 0.8800 | 0.8797 | 0.8780 |
| 12.4527 | 1850 | 16.5467 | 16.3991 | 0.8796 | 0.8806 | 0.8804 | 0.8790 |
| 12.7892 | 1900 | 16.5398 | 16.3970 | 0.8792 | 0.8798 | 0.8801 | 0.8788 |
| 13.1258 | 1950 | 16.5047 | 16.3964 | 0.8796 | 0.8804 | 0.8804 | 0.8788 |
| 13.4623 | 2000 | 16.4985 | 16.4025 | 0.8793 | 0.8798 | 0.8807 | 0.8790 |
| 13.7989 | 2050 | 16.4852 | 16.4107 | 0.8801 | 0.8810 | 0.8800 | 0.8793 |
| 14.1355 | 2100 | 16.4526 | 16.3929 | 0.8797 | 0.8801 | 0.8809 | 0.8779 |
| 14.4720 | 2150 | 16.4343 | 16.4075 | 0.8788 | 0.8791 | 0.8797 | 0.8774 |
| 14.8086 | 2200 | 16.4244 | 16.4027 | 0.8804 | 0.8819 | 0.8820 | 0.8809 |
| 15.1451 | 2250 | 16.3947 | 16.4102 | 0.8791 | 0.8792 | 0.8803 | 0.8773 |
| 15.4817 | 2300 | 16.3827 | 16.4042 | 0.8804 | 0.8813 | 0.8813 | 0.8781 |
| 15.8183 | 2350 | 16.3719 | 16.4003 | 0.8801 | 0.8818 | 0.8820 | 0.8791 |
| 16.1548 | 2400 | 16.3403 | 16.4132 | 0.8781 | 0.8791 | 0.8799 | 0.8767 |
| 16.4914 | 2450 | 16.3357 | 16.4149 | 0.8804 | 0.8809 | 0.8807 | 0.8792 |
| 16.8279 | 2500 | 16.3203 | 16.4081 | 0.8804 | 0.8814 | 0.8816 | 0.8791 |
| 17.1645 | 2550 | 16.2986 | 16.4139 | 0.8798 | 0.8800 | 0.8820 | 0.8791 |
| 17.5011 | 2600 | 16.2923 | 16.4062 | 0.8786 | 0.8792 | 0.8799 | 0.8768 |
| 17.8376 | 2650 | 16.2649 | 16.4106 | 0.8800 | 0.8807 | 0.8814 | 0.8787 |
| 18.1742 | 2700 | 16.2505 | 16.4188 | 0.8786 | 0.8793 | 0.8803 | 0.8771 |
| 18.5107 | 2750 | 16.226 | 16.4149 | 0.8771 | 0.8781 | 0.8780 | 0.8766 |
| 18.8473 | 2800 | 16.2106 | 16.4230 | 0.8780 | 0.8799 | 0.8791 | 0.8767 |
| 19.1838 | 2850 | 16.2052 | 16.4351 | 0.8770 | 0.8777 | 0.8785 | 0.8745 |
| 19.5204 | 2900 | 16.186 | 16.4331 | 0.8777 | 0.8793 | 0.8792 | 0.8762 |
| 19.8570 | 2950 | 16.1496 | 16.4377 | 0.8774 | 0.8781 | 0.8780 | 0.8771 |
| 20.1935 | 3000 | 16.151 | 16.4407 | 0.8766 | 0.8780 | 0.8780 | 0.8751 |
| 20.5301 | 3050 | 16.1081 | 16.4426 | 0.8759 | 0.8775 | 0.8774 | 0.8749 |
| 20.8666 | 3100 | 16.0864 | 16.4412 | 0.8774 | 0.8781 | 0.8787 | 0.8746 |
| 21.2032 | 3150 | 16.0934 | 16.4547 | 0.8768 | 0.8783 | 0.8794 | 0.8746 |
| 21.5398 | 3200 | 16.0382 | 16.4589 | 0.8742 | 0.8752 | 0.8766 | 0.8723 |
| 21.8763 | 3250 | 16.0279 | 16.4668 | 0.8752 | 0.8766 | 0.8773 | 0.8728 |
| 22.2129 | 3300 | 16.0327 | 16.4737 | 0.8742 | 0.8768 | 0.8773 | 0.8727 |
| 22.5494 | 3350 | 15.979 | 16.4686 | 0.8740 | 0.8771 | 0.8771 | 0.8722 |
| 22.8860 | 3400 | 15.9622 | 16.4736 | 0.8743 | 0.8760 | 0.8765 | 0.8721 |
| 23.2225 | 3450 | 15.9881 | 16.4802 | 0.8743 | 0.8757 | 0.8755 | 0.8723 |
| 23.5591 | 3500 | 15.9482 | 16.4821 | 0.8725 | 0.8761 | 0.8761 | 0.8710 |
| 23.8957 | 3550 | 15.9228 | 16.4996 | 0.8726 | 0.8748 | 0.8751 | 0.8709 |
| 24.2322 | 3600 | 15.9418 | 16.4973 | 0.8709 | 0.8728 | 0.8734 | 0.8699 |
| 24.5688 | 3650 | 15.896 | 16.4985 | 0.8696 | 0.8716 | 0.8727 | 0.8686 |
| 24.9053 | 3700 | 15.8788 | 16.5172 | 0.8691 | 0.8715 | 0.8717 | 0.8662 |
| 25.2419 | 3750 | 15.9147 | 16.5062 | 0.8677 | 0.8706 | 0.8712 | 0.8662 |
| 25.5785 | 3800 | 15.857 | 16.5058 | 0.8683 | 0.8717 | 0.8732 | 0.8663 |
| 25.9150 | 3850 | 15.8291 | 16.5207 | 0.8674 | 0.8702 | 0.8706 | 0.8644 |
| 26.2516 | 3900 | 15.8802 | 16.5233 | 0.8678 | 0.8697 | 0.8714 | 0.8664 |
| 26.5881 | 3950 | 15.846 | 16.5170 | 0.8686 | 0.8713 | 0.8717 | 0.8655 |
| 26.9247 | 4000 | 15.8012 | 16.5336 | 0.8663 | 0.8682 | 0.8699 | 0.8635 |
### Framework Versions
- Python: 3.10.10
- Sentence Transformers: 3.0.1
- Transformers: 4.41.2
- PyTorch: 2.1.2+cu121
- Accelerate: 0.33.0
- Datasets: 2.19.1
- Tokenizers: 0.19.1
## Citation
### BibTeX
#### Sentence Transformers
```bibtex
@inproceedings{reimers-2019-sentence-bert,
title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
author = "Reimers, Nils and Gurevych, Iryna",
booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
month = "11",
year = "2019",
publisher = "Association for Computational Linguistics",
url = "https://arxiv.org/abs/1908.10084",
}
```
#### MatryoshkaLoss
```bibtex
@misc{kusupati2024matryoshka,
title={Matryoshka Representation Learning},
author={Aditya Kusupati and Gantavya Bhatt and Aniket Rege and Matthew Wallingford and Aditya Sinha and Vivek Ramanujan and William Howard-Snyder and Kaifeng Chen and Sham Kakade and Prateek Jain and Ali Farhadi},
year={2024},
eprint={2205.13147},
archivePrefix={arXiv},
primaryClass={cs.LG}
}
```
#### TripletLoss
```bibtex
@misc{hermans2017defense,
title={In Defense of the Triplet Loss for Person Re-Identification},
author={Alexander Hermans and Lucas Beyer and Bastian Leibe},
year={2017},
eprint={1703.07737},
archivePrefix={arXiv},
primaryClass={cs.CV}
}
```
<!--
## Glossary
*Clearly define terms in order to be accessible across audiences.*
-->
<!--
## Model Card Authors
*Lists the people who create the model card, providing recognition and accountability for the detailed work that goes into its construction.*
-->
<!--
## Model Card Contact
*Provides a way for people who have updates to the Model Card, suggestions, or questions, to contact the Model Card authors.*
--> |