rajatrc1705's picture
model at final check point
1663fca
metadata
base_model: BAAI/bge-small-en
datasets:
  - sentence-transformers/hotpotqa
language:
  - en
library_name: sentence-transformers
license: apache-2.0
metrics:
  - cosine_accuracy
  - dot_accuracy
  - manhattan_accuracy
  - euclidean_accuracy
  - max_accuracy
pipeline_tag: sentence-similarity
tags:
  - sentence-transformers
  - sentence-similarity
  - feature-extraction
  - generated_from_trainer
  - dataset_size:76064
  - loss:MatryoshkaLoss
  - loss:TripletLoss
widget:
  - source_sentence: >-
      When was the brewery founded by the family who resided in the Lemp Mansion
      founded?
    sentences:
      - >-
        Latrobe Brewing Company Latrobe Brewing Company in Latrobe,
        Pennsylvania, founded in 1839, was one of the largest breweries in the
        United States and the maker of "Rolling Rock" beer (famous for its small
        green bottles). It was purchased by Labatt Brewing Company in 1987,
        which in turn was purchased in 1995 by the Belgian brewing conglomerate
        corporation Interbrew, which merged later into InBev in 2004.
      - >-
        Lemp Mansion The Lemp Mansion (3322 DeMenil Place, St. Louis, Missouri)
        is a historical house in Benton Park, St. Louis, Missouri. It is also
        the site of four suicides by Lemp family members after the death of the
        son Frederick Lemp, whose William J. Lemp Brewing Co. dominated the St.
        Louis beer market before Prohibition with its Falstaff beer brand. The
        mansion is said to be haunted by members of the Lemp family.
      - >-
        Man of the House (song) "Man of the House" is a song co-written and
        recorded by American country music artist Chuck Wicks. It was released
        in January 2009 as the third single from the album "Starting Now". The
        song reached #27 on the "Billboard" Hot Country Songs chart. The song
        was written by Wicks and Michael Mobley.
  - source_sentence: >-
      Who is the translator and researcher who served as a dean at Hartford
      Junior College and translated her Belgian-French partner's 1968 work into
      English?
    sentences:
      - >-
        William L. Shirer William Lawrence Shirer (February 23, 1904 – December
        28, 1993) was an American journalist and war correspondent. He wrote
        "The Rise and Fall of the Third Reich", a history of Nazi Germany that
        has been read by many and cited in scholarly works for more than 50
        years. Originally a foreign correspondent for the "Chicago Tribune" and
        the International News Service, Shirer was the first reporter hired by
        Edward R. Murrow for what would become a CBS radio team of journalists
        known as "Murrow's Boys". He became known for his broadcasts from
        Berlin, from the rise of the Nazi dictatorship through the first year of
        World War II (1940). With Murrow, he organized the first broadcast world
        news roundup, a format still followed by news broadcasts.
      - >-
        The Abyss (Yourcenar novel) The Abyss (French: L'Œuvre au noir ) is a
        1968 novel by the Belgian-French writer Marguerite Yourcenar. Its
        narrative centers on the life and death of Zeno, a physician,
        philosopher, scientist and alchemist born in Bruges during the
        Renaissance era. The book was published in France in 1968 and was met
        with immediate popular interest as well as critical acclaim, obtaining
        the Prix Femina with unanimous votes the year of its publication. The
        English translation by Grace Frick has been published under the title
        "The Abyss" or alternatively Zeno of Bruges. Belgian filmmaker André
        Delvaux adapted it into a film in 1988.
      - >-
        Pierre L. van den Berghe Pierre L. van den Berghe (born 1933) is
        professor emeritus of sociology and anthropology at the University of
        Washington, where he has worked since 1965. Born in the Belgian Congo to
        Belgian parents, and spending World War II in occupied Belgium, he was
        an early witness to ethnic conflict and racism, which eventually led him
        to become a leading authority on ethnic relations. He has conducted
        field work in South Africa, Mexico, Guatemala, Iran, Lebanon, Nigeria,
        Peru, and Israel. Early in his career, he lectured at the University of
        Natal alongside Leo Kuper and Fatima Meer. A student of Talcott Parsons
        at Harvard (receiving the Ph.D. in 1960), he nevertheless had little
        interest in structural functionalism and was one of the first proponents
        of sociobiological approaches to social phenomena.
  - source_sentence: Them Crooked Vultures and The Vines are both part of what music genres?
    sentences:
      - >-
        Friendly Center Friendly Center is a large, open-air shopping center
        located in northwestern Greensboro, North Carolina, near the
        intersection of Wendover Avenue and Friendly Avenue. The shopping center
        opened in August 1957, and with its inward orientation, Friendly Center
        could be classified as an outdoor lifestyle center. Its anchor tenants
        include Belk, Macy's, and Sears. Other tenants include Barnes & Noble,
        Old Navy, The Grande Theatre is a 16-screen multiplex cinema operated by
        Regal Cinemas. It also contains Harris Teeter's flagship supermarket
        location encompassing 72,000 square feet (6,700 m2) and Whole Foods
        Market. There are specialty "foodie" stores tucked away in the back
        corner by Harris Teeter such as the Savory Spice Shop and Midtown Olive
        Oil. It features a number of national retailers such as Banana Republic,
        Victoria's Secret, The Limited, Bath & Body Works, Express, The GAP,
        Eddie Bauer, Talbots, Birkenstock Feet First, Pier 1, and New York &
        Company.
      - >-
        Twin Wild Twin Wild is a British four-piece alternative rock band.
        Formed in 2012, the band is made up of the collective creative energies
        of Richard Hutchison (vocals, guitars), Imran Mair (drums), David Cuzner
        (guitars) and Edward Thomas (bass). In 2014, the band self-released
        their track "Fears", which garnered over half a million plays on
        Soundcloud and charted in Spotify’s Top 20 viral chart. Their style of
        music has been compared to the likes of Foals, The Neighbourhood, and
        Bastille. The band have been hailed by Edith Bowman as "The love child
        of Bastille and Biffy Clyro".
      - >-
        Them Crooked Vultures Them Crooked Vultures is a rock supergroup formed
        in Los Angeles in 2009 by John Paul Jones (former member of Led
        Zeppelin) on bass and keyboards, Dave Grohl (of Foo Fighters and
        formerly of Nirvana) on drums and backing vocals, and Josh Homme (of
        Queens of the Stone Age, Eagles of Death Metal and formerly of Kyuss) on
        guitar and vocals. The group also includes guitarist Alain Johannes
        during live performances. The band began recording in February 2009, and
        performed their first gig on August 9, 2009, in Chicago, followed by a
        European debut on August 19. On October 1 the group embarked on a
        worldwide tour titled "Deserve the Future" with dates going into 2010.
        The band's first single "New Fang" was released in October 2009,
        followed by the group's self-titled debut album the following month,
        debuting at number 12 on the "Billboard" 200. The group won the 2011
        Grammy Award for Best Hard Rock Performance for "New Fang".
  - source_sentence: >-
      What type of collection does Nådens år and Agnetha Fältskog have in
      common?
    sentences:
      - >-
        Gardiner Island (Nunavut) Gardiner Island is one of the many uninhabited
        Canadian arctic islands in Qikiqtaaluk Region, Nunavut. It is a Baffin
        Island offshore island located in Frobisher Bay south of the capital
        city of Iqaluit.
      - >-
        Nådens år Nådens år (The Year of Grace) is the third studio album by the
        Swedish rock artist Ulf Lundell. It was released in April 1978 on EMI
        and Parlophone. It was recorded in EMI Studio, Stockholm, and produced
        by Kjell Andersson and Lundell. It includes "Snön faller och vi med den"
        ("The snow is falling and we are too"), one of Lundell's more famous
        songs. Agnetha Fältskog is involved in the song. The cover picture shows
        Lundell sitting on a rock next to a dog and was taken in Åre in 1977.
        "Nådens år" achieved Gold status in Sweden.
      - >-
        Åsa Elzén Åsa Elzén is an artist whose work is informed by feminist
        theory, intersectionality and post-colonialism. Elzén was born in Sweden
        in 1972 and currently lives and works in Berlin.
  - source_sentence: >-
      Red Velvet is a 2012 play by Lolita Chakrabarti, dealing with the
      biography of a 19th century actor born in which year ?
    sentences:
      - >-
        Herbert Campbell Herbert Campbell (22 December 1844 – 19 July 1904) born
        Herbert Edward Story was an English comedian and actor who appeared in
        music hall, Victorian burlesques and musical comedies during the
        Victorian era. He was famous for starring, for forty years, in the
        Theatre Royal, Drury Lane's annual Christmas pantomimes, predominantly
        as a dame.
      - >-
        Leptinella Leptinella is a genus of alpine flowering plant in the
        Asteraceae family, comprising 33 species, distributed in New Guinea,
        Australia, New Zealand, South Africa, and South America. Many of the
        species are endemic to New Zealand.
      - >-
        Red Velvet (play) Red Velvet is a 2012 play by Lolita Chakrabarti,
        dealing with the biography of the 19th century actor Ira Aldridge and
        his taking the role of "Othello". It premiered at the Tricycle Theatre
        (directed by its new artistic director Indhu Rubasingham) from 11
        October to 24 November 2012, with Aldridge played by Adrian Lester.
model-index:
  - name: BGE-base-en-v1.5-Hotpotqa
    results:
      - task:
          type: triplet
          name: Triplet
        dataset:
          name: dim 384
          type: dim_384
        metrics:
          - type: cosine_accuracy
            value: 0.8698532891623284
            name: Cosine Accuracy
          - type: dot_accuracy
            value: 0.13014671083767157
            name: Dot Accuracy
          - type: manhattan_accuracy
            value: 0.8737576904874585
            name: Manhattan Accuracy
          - type: euclidean_accuracy
            value: 0.8698532891623284
            name: Euclidean Accuracy
          - type: max_accuracy
            value: 0.8737576904874585
            name: Max Accuracy
      - task:
          type: triplet
          name: Triplet
        dataset:
          name: dim 256
          type: dim_256
        metrics:
          - type: cosine_accuracy
            value: 0.86819687647894
            name: Cosine Accuracy
          - type: dot_accuracy
            value: 0.1319214387127307
            name: Dot Accuracy
          - type: manhattan_accuracy
            value: 0.8762423095125415
            name: Manhattan Accuracy
          - type: euclidean_accuracy
            value: 0.8680785612872692
            name: Euclidean Accuracy
          - type: max_accuracy
            value: 0.8762423095125415
            name: Max Accuracy
      - task:
          type: triplet
          name: Triplet
        dataset:
          name: dim 128
          type: dim_128
        metrics:
          - type: cosine_accuracy
            value: 0.8663038334122102
            name: Cosine Accuracy
          - type: dot_accuracy
            value: 0.1386654046379555
            name: Dot Accuracy
          - type: manhattan_accuracy
            value: 0.8742309512541411
            name: Manhattan Accuracy
          - type: euclidean_accuracy
            value: 0.865712257453857
            name: Euclidean Accuracy
          - type: max_accuracy
            value: 0.8742309512541411
            name: Max Accuracy
      - task:
          type: triplet
          name: Triplet
        dataset:
          name: dim 64
          type: dim_64
        metrics:
          - type: cosine_accuracy
            value: 0.8634642688121155
            name: Cosine Accuracy
          - type: dot_accuracy
            value: 0.1508518693800284
            name: Dot Accuracy
          - type: manhattan_accuracy
            value: 0.8728111689540937
            name: Manhattan Accuracy
          - type: euclidean_accuracy
            value: 0.8637008991954567
            name: Euclidean Accuracy
          - type: max_accuracy
            value: 0.8728111689540937
            name: Max Accuracy

BGE-base-en-v1.5-Hotpotqa

This is a sentence-transformers model finetuned from BAAI/bge-small-en on the sentence-transformers/hotpotqa dataset. It maps sentences & paragraphs to a 384-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

  • Model Type: Sentence Transformer
  • Base model: BAAI/bge-small-en
  • Maximum Sequence Length: 512 tokens
  • Output Dimensionality: 384 tokens
  • Similarity Function: Cosine Similarity
  • Training Dataset:
  • Language: en
  • License: apache-2.0

Model Sources

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 512, 'do_lower_case': True}) with Transformer model: BertModel 
  (1): Pooling({'word_embedding_dimension': 384, 'pooling_mode_cls_token': True, 'pooling_mode_mean_tokens': False, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
  (2): Normalize()
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("sentence_transformers_model_id")
# Run inference
sentences = [
    'Red Velvet is a 2012 play by Lolita Chakrabarti, dealing with the biography of a 19th century actor born in which year ?',
    'Red Velvet (play) Red Velvet is a 2012 play by Lolita Chakrabarti, dealing with the biography of the 19th century actor Ira Aldridge and his taking the role of "Othello". It premiered at the Tricycle Theatre (directed by its new artistic director Indhu Rubasingham) from 11 October to 24 November 2012, with Aldridge played by Adrian Lester.',
    "Herbert Campbell Herbert Campbell (22 December 1844 – 19 July 1904) born Herbert Edward Story was an English comedian and actor who appeared in music hall, Victorian burlesques and musical comedies during the Victorian era. He was famous for starring, for forty years, in the Theatre Royal, Drury Lane's annual Christmas pantomimes, predominantly as a dame.",
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 384]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

Evaluation

Metrics

Triplet

Metric Value
cosine_accuracy 0.8699
dot_accuracy 0.1301
manhattan_accuracy 0.8738
euclidean_accuracy 0.8699
max_accuracy 0.8738

Triplet

Metric Value
cosine_accuracy 0.8682
dot_accuracy 0.1319
manhattan_accuracy 0.8762
euclidean_accuracy 0.8681
max_accuracy 0.8762

Triplet

Metric Value
cosine_accuracy 0.8663
dot_accuracy 0.1387
manhattan_accuracy 0.8742
euclidean_accuracy 0.8657
max_accuracy 0.8742

Triplet

Metric Value
cosine_accuracy 0.8635
dot_accuracy 0.1509
manhattan_accuracy 0.8728
euclidean_accuracy 0.8637
max_accuracy 0.8728

Training Details

Training Dataset

sentence-transformers/hotpotqa

  • Dataset: sentence-transformers/hotpotqa at f07d3cd
  • Size: 76,064 training samples
  • Columns: anchor, positive, and negative
  • Approximate statistics based on the first 1000 samples:
    anchor positive negative
    type string string string
    details
    • min: 8 tokens
    • mean: 24.08 tokens
    • max: 95 tokens
    • min: 23 tokens
    • mean: 100.12 tokens
    • max: 512 tokens
    • min: 15 tokens
    • mean: 88.02 tokens
    • max: 512 tokens
  • Samples:
    anchor positive negative
    What nationality was the player named MVP in 2017 World Baseball Classic – Pool C ? 2017 World Baseball Classic – Pool C Pool C of the First Round of the 2017 World Baseball Classic was held at Marlins Park, Miami, Florida, United States, from March 9 to 12, 2017, between Canada, Colombia, the Dominican Republic, and the United States. Pool C was a round-robin tournament. Each team played the other three teams once, with the top two teams – the Dominican Republic and the United States – advancing to Pool F, one of two second-round pools. Manny Machado of the Dominican Republic was named MVP for the first-round Pool C bracket of the WBC, after batting .357. 2017 World Baseball Classic – Qualifier 2 Qualifier 2 of the Qualifying Round of the 2017 World Baseball Classic was held at Estadio B'Air, Mexicali, Mexico from March 17 to 20, 2016.
    Karl Kraepelin specialized in the study of what predatory arachnids? Karl Kraepelin Karl Matthias Friedrich Magnus Kraepelin (14 December 1848 Neustrelitz – 28 June 1915 Hamburg), was a German naturalist who specialised in the study of scorpions, centipedes, spiders and solfugids, and was noted for his monograph ""Scorpiones und Pedipalpi"" (Berlin) in 1899, which was an exhaustive survey of the taxonomy of the Order Scorpiones. From 1889–1914 he was Director of the "Naturhistorisches Museum Hamburg ", which was destroyed during World War II, and worked on myriapods from 1901–1916. Teuthology Teuthology (from Greek "τεῦθος" , "cuttlefish, squid", and -λογία , "-logia") is the study of cephalopods.
    Who directed the 1990 American crime film in which Vito Pick me played a bit part? Vito Picone Vito Picone (born March 20, 1941) is the lead singer of The Elegants, and along with Jimmy Mochella is a remaining original member. He has also played bit parts in "Goodfellas", "Analyze This", and "The Sopranos". The Rookie (1990 film) The Rookie is a 1990 American buddy cop film directed by Clint Eastwood and produced by Howard G. Kazanjian, Steven Siebert and David Valdes. It was written from a screenplay conceived by Boaz Yakin and Scott Spiegel. The film stars Charlie Sheen, Clint Eastwood, Raúl Juliá, Sônia Braga, Lara Flynn Boyle, and Tom Skerritt. Eastwood plays a veteran police officer teamed up with a younger detective played by Sheen ("the rookie"), whose intent is to take down a German crime lord in downtown Los Angeles following months of investigation into an exotic car theft ring.
  • Loss: MatryoshkaLoss with these parameters:
    {
        "loss": "TripletLoss",
        "matryoshka_dims": [
            384,
            256,
            128,
            64
        ],
        "matryoshka_weights": [
            1,
            1,
            1,
            1
        ],
        "n_dims_per_step": -1
    }
    

Evaluation Dataset

sentence-transformers/hotpotqa

  • Dataset: sentence-transformers/hotpotqa at f07d3cd
  • Size: 8,452 evaluation samples
  • Columns: anchor, positive, and negative
  • Approximate statistics based on the first 1000 samples:
    anchor positive negative
    type string string string
    details
    • min: 10 tokens
    • mean: 24.53 tokens
    • max: 114 tokens
    • min: 19 tokens
    • mean: 103.87 tokens
    • max: 407 tokens
    • min: 17 tokens
    • mean: 88.32 tokens
    • max: 356 tokens
  • Samples:
    anchor positive negative
    Which actress, known for her role as Harper Munroe on the MTV comedy series "Happyland", starred alongside Laura Marano, Parker Mack, Michelle Clunie and Kathleen Wilhoite in the film A Sort of Homecoming? A Sort of Homecoming (film) A Sort of Homecoming is an American drama directed by Maria Burton, her fifth feature film. The films stars Katherine McNamara, Laura Marano, Parker Mack, Michelle Clunie and Kathleen Wilhoite. The film premiered March 14, 2015 at the Omaha Film Festival. Nellie Bellflower Nellie Bellflower (born May 1, 1946 in Phoenix, Arizona) is an American actress and voice artist who provided the voice of Princess Ariel in the Ruby-Spears animated television series "Thundarr the Barbarian". She has also been in "The Last Unicorn" (voice), Rankin/Bass "The Return of the King", "Americathon", the miniseries "East of Eden", and guest roles on various TV shows such as "Barnaby Jones", "Barney Miller", "Starsky and Hutch", and "Happy Days" as Fonzie's ex-fiancée Maureen Johnson, a.k.a. "The Lone Stripper", in the Season 2 episode of the series titled "Fonzie's Getting Married" (episode #13). Nellie has been involved in movie production with three projects: "The Girl in Melanie Klein" (2008), "Miss Pettigrew Lives for a Day" (2008) and "Finding Neverland" (2004), for which she was nominated for an Academy Award as Producer for Best Picture. She is married to Michael Mislove.
    Between Pizza Fusion and Pizzeria Venti, which restaurant emphasizes organic ingredients and green building methods? Pizza Fusion Pizza Fusion is a Deerfield Beach, Florida-based pizza restaurant chain. Using mostly organic ingredients and emphasizing green building methods, the restaurants operate under the tagline Saving the Earth, One Pizza at a Time. Pizza Schmizza Pizza Schmizza is an American pizza chain with 23 locations throughout the Portland, Oregon area, and two in southern Oregon. Pizza Schmizza, primarily selling thin crust pizza by-the-slice.
    What company distributed the stop motion spin-off special "The Year Without a Santa Claus," which aired on December 10, 1974? A Miser Brothers' Christmas A Miser Brothers' Christmas is a stop motion spin-off special based on some of the characters from the 1974 Rankin-Bass special "The Year Without a Santa Claus". Distributed by Warner Bros. Animation under their Warner Premiere label (the rights holders of the post-1974 Rankin-Bass library) and Toronto-based Cuppa Coffee Studios, the one-hour special premiered on ABC Family on Saturday, December 13, 2008, during the network's annual The 25 Days of Christmas programming. Mickey Rooney and George S. Irving reprised their respective roles as Santa Claus and Heat Miser at ages 88 and 86. Snow Miser, originally portrayed by Dick Shawn who died in 1987, was voiced by Juan Chioran, while Mrs. Claus, voiced by Shirley Booth in the original, was portrayed by Catherine Disher (because Booth had died in 1992). The movie aimed to emulate the Rankin/Bass animation style. This is the last Christmas special to feature Mickey Rooney as Santa Claus, as he died in 2014, as well as the last time George Irving voiced Heat Miser, as he died in 2016. Holidaze: The Christmas That Almost Didn't Happen Holidaze: The Christmas That Almost Didn't Happen is an American stop-motion animated Christmas television special directed by David H. Brooks, that originally aired in 2006 and produced by BixPix Entertainment, Once Upon a Frog and Madison Street Entertainment. The show's plot has Rusty Reindeer (Fred Savage) the brother of Rudolph the Red Nosed Reindeer joining a support group for depressed holiday icons, and he and the other characters search for the meaning of Christmas and help a young boy (Dylan and Cole Sprouse) to get on Santa's nice list. Rusty's cohorts include Candie, the Easter Bunny (Gladys Knight); Mr. C, the grouchy cherub (Paul Rodriguez); Albert, the Thanksgiving Turkey (Harland Williams); And Trick and Treat (Brenda Song and Emily Osment) the teenage Halloween Ghosts.
  • Loss: MatryoshkaLoss with these parameters:
    {
        "loss": "TripletLoss",
        "matryoshka_dims": [
            384,
            256,
            128,
            64
        ],
        "matryoshka_weights": [
            1,
            1,
            1,
            1
        ],
        "n_dims_per_step": -1
    }
    

Training Hyperparameters

Non-Default Hyperparameters

  • eval_strategy: steps
  • per_device_train_batch_size: 32
  • per_device_eval_batch_size: 32
  • gradient_accumulation_steps: 16
  • learning_rate: 2e-05
  • num_train_epochs: 50
  • lr_scheduler_type: cosine
  • warmup_ratio: 0.1
  • bf16: True
  • tf32: True
  • load_best_model_at_end: True
  • optim: adamw_torch_fused
  • resume_from_checkpoint: bge-small-hotpotwa-matryoshka
  • batch_sampler: no_duplicates

All Hyperparameters

Click to expand
  • overwrite_output_dir: False
  • do_predict: False
  • eval_strategy: steps
  • prediction_loss_only: True
  • per_device_train_batch_size: 32
  • per_device_eval_batch_size: 32
  • per_gpu_train_batch_size: None
  • per_gpu_eval_batch_size: None
  • gradient_accumulation_steps: 16
  • eval_accumulation_steps: None
  • learning_rate: 2e-05
  • weight_decay: 0.0
  • adam_beta1: 0.9
  • adam_beta2: 0.999
  • adam_epsilon: 1e-08
  • max_grad_norm: 1.0
  • num_train_epochs: 50
  • max_steps: -1
  • lr_scheduler_type: cosine
  • lr_scheduler_kwargs: {}
  • warmup_ratio: 0.1
  • warmup_steps: 0
  • log_level: passive
  • log_level_replica: warning
  • log_on_each_node: True
  • logging_nan_inf_filter: True
  • save_safetensors: True
  • save_on_each_node: False
  • save_only_model: False
  • restore_callback_states_from_checkpoint: False
  • no_cuda: False
  • use_cpu: False
  • use_mps_device: False
  • seed: 42
  • data_seed: None
  • jit_mode_eval: False
  • use_ipex: False
  • bf16: True
  • fp16: False
  • fp16_opt_level: O1
  • half_precision_backend: auto
  • bf16_full_eval: False
  • fp16_full_eval: False
  • tf32: True
  • local_rank: 0
  • ddp_backend: None
  • tpu_num_cores: None
  • tpu_metrics_debug: False
  • debug: []
  • dataloader_drop_last: False
  • dataloader_num_workers: 0
  • dataloader_prefetch_factor: None
  • past_index: -1
  • disable_tqdm: False
  • remove_unused_columns: True
  • label_names: None
  • load_best_model_at_end: True
  • ignore_data_skip: False
  • fsdp: []
  • fsdp_min_num_params: 0
  • fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
  • fsdp_transformer_layer_cls_to_wrap: None
  • accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
  • deepspeed: None
  • label_smoothing_factor: 0.0
  • optim: adamw_torch_fused
  • optim_args: None
  • adafactor: False
  • group_by_length: False
  • length_column_name: length
  • ddp_find_unused_parameters: None
  • ddp_bucket_cap_mb: None
  • ddp_broadcast_buffers: False
  • dataloader_pin_memory: True
  • dataloader_persistent_workers: False
  • skip_memory_metrics: True
  • use_legacy_prediction_loop: False
  • push_to_hub: False
  • resume_from_checkpoint: bge-small-hotpotwa-matryoshka
  • hub_model_id: None
  • hub_strategy: every_save
  • hub_private_repo: False
  • hub_always_push: False
  • gradient_checkpointing: False
  • gradient_checkpointing_kwargs: None
  • include_inputs_for_metrics: False
  • eval_do_concat_batches: True
  • fp16_backend: auto
  • push_to_hub_model_id: None
  • push_to_hub_organization: None
  • mp_parameters:
  • auto_find_batch_size: False
  • full_determinism: False
  • torchdynamo: None
  • ray_scope: last
  • ddp_timeout: 1800
  • torch_compile: False
  • torch_compile_backend: None
  • torch_compile_mode: None
  • dispatch_batches: None
  • split_batches: None
  • include_tokens_per_second: False
  • include_num_input_tokens_seen: False
  • neftune_noise_alpha: None
  • optim_target_modules: None
  • batch_eval_metrics: False
  • batch_sampler: no_duplicates
  • multi_dataset_batch_sampler: proportional

Training Logs

Epoch Step Training Loss loss dim_128_cosine_accuracy dim_256_cosine_accuracy dim_384_cosine_accuracy dim_64_cosine_accuracy
0.3366 50 19.5758 19.3933 0.9552 0.9663 0.9668 0.9359
0.6731 100 19.4573 19.0971 0.9571 0.9646 0.9653 0.9450
1.0097 150 19.1409 18.4070 0.9385 0.9434 0.9473 0.9307
1.3462 200 18.6431 17.3292 0.9126 0.9164 0.9184 0.9094
1.6828 250 18.2288 16.8751 0.9063 0.9071 0.9100 0.9023
2.0194 300 18.0425 16.6981 0.9020 0.9032 0.9045 0.8990
2.3559 350 17.9458 16.6155 0.9037 0.9013 0.9022 0.8984
2.6925 400 17.8525 16.5536 0.8978 0.8971 0.8974 0.8948
3.0290 450 17.7529 16.5136 0.8980 0.8956 0.8953 0.8951
3.3656 500 17.6709 16.4824 0.8932 0.8914 0.8928 0.8907
3.7021 550 17.5348 16.4632 0.8863 0.8858 0.8859 0.8849
4.0387 600 17.4198 16.4601 0.8852 0.8862 0.8859 0.8839
4.3753 650 17.3673 16.4405 0.8854 0.8864 0.8865 0.8842
4.7118 700 17.2603 16.4356 0.8835 0.8838 0.8838 0.8807
5.0484 750 17.1807 16.4443 0.8850 0.8864 0.8859 0.8838
5.3849 800 17.1629 16.4202 0.8848 0.8862 0.8867 0.8842
5.7215 850 17.0747 16.4162 0.8854 0.8875 0.8869 0.8837
6.0581 900 17.0161 16.4192 0.8852 0.8863 0.8856 0.8856
6.3946 950 17.0146 16.4033 0.8849 0.8854 0.8856 0.8844
6.7312 1000 16.9393 16.4053 0.8829 0.8839 0.8848 0.8835
7.0677 1050 16.899 16.4162 0.8826 0.8829 0.8833 0.8818
7.4043 1100 16.9112 16.4051 0.8829 0.8835 0.8833 0.8820
7.7408 1150 16.8508 16.4044 0.8822 0.8825 0.8830 0.8820
8.0774 1200 16.8104 16.4063 0.8816 0.8816 0.8814 0.8817
8.4140 1250 16.8212 16.4040 0.8835 0.8822 0.8822 0.8820
8.7505 1300 16.7743 16.3934 0.8822 0.8824 0.8817 0.8810
9.0871 1350 16.7383 16.3963 0.8810 0.8820 0.8807 0.8800
9.4236 1400 16.743 16.4067 0.8819 0.8822 0.8819 0.8798
9.7602 1450 16.7047 16.3959 0.8804 0.8810 0.8810 0.8797
10.0968 1500 16.6782 16.3986 0.8788 0.8791 0.8796 0.8784
10.4333 1550 16.6708 16.4016 0.8794 0.8792 0.8797 0.8791
10.7699 1600 16.6485 16.3963 0.8790 0.8801 0.8791 0.8781
11.1064 1650 16.6205 16.4012 0.8779 0.8787 0.8793 0.8771
11.4430 1700 16.6095 16.4131 0.8786 0.8790 0.8794 0.8791
11.7796 1750 16.5891 16.4070 0.8807 0.8805 0.8810 0.8801
12.1161 1800 16.5619 16.3963 0.8794 0.8800 0.8797 0.8780
12.4527 1850 16.5467 16.3991 0.8796 0.8806 0.8804 0.8790
12.7892 1900 16.5398 16.3970 0.8792 0.8798 0.8801 0.8788
13.1258 1950 16.5047 16.3964 0.8796 0.8804 0.8804 0.8788
13.4623 2000 16.4985 16.4025 0.8793 0.8798 0.8807 0.8790
13.7989 2050 16.4852 16.4107 0.8801 0.8810 0.8800 0.8793
14.1355 2100 16.4526 16.3929 0.8797 0.8801 0.8809 0.8779
14.4720 2150 16.4343 16.4075 0.8788 0.8791 0.8797 0.8774
14.8086 2200 16.4244 16.4027 0.8804 0.8819 0.8820 0.8809
15.1451 2250 16.3947 16.4102 0.8791 0.8792 0.8803 0.8773
15.4817 2300 16.3827 16.4042 0.8804 0.8813 0.8813 0.8781
15.8183 2350 16.3719 16.4003 0.8801 0.8818 0.8820 0.8791
16.1548 2400 16.3403 16.4132 0.8781 0.8791 0.8799 0.8767
16.4914 2450 16.3357 16.4149 0.8804 0.8809 0.8807 0.8792
16.8279 2500 16.3203 16.4081 0.8804 0.8814 0.8816 0.8791
17.1645 2550 16.2986 16.4139 0.8798 0.8800 0.8820 0.8791
17.5011 2600 16.2923 16.4062 0.8786 0.8792 0.8799 0.8768
17.8376 2650 16.2649 16.4106 0.8800 0.8807 0.8814 0.8787
18.1742 2700 16.2505 16.4188 0.8786 0.8793 0.8803 0.8771
18.5107 2750 16.226 16.4149 0.8771 0.8781 0.8780 0.8766
18.8473 2800 16.2106 16.4230 0.8780 0.8799 0.8791 0.8767
19.1838 2850 16.2052 16.4351 0.8770 0.8777 0.8785 0.8745
19.5204 2900 16.186 16.4331 0.8777 0.8793 0.8792 0.8762
19.8570 2950 16.1496 16.4377 0.8774 0.8781 0.8780 0.8771
20.1935 3000 16.151 16.4407 0.8766 0.8780 0.8780 0.8751
20.5301 3050 16.1081 16.4426 0.8759 0.8775 0.8774 0.8749
20.8666 3100 16.0864 16.4412 0.8774 0.8781 0.8787 0.8746
21.2032 3150 16.0934 16.4547 0.8768 0.8783 0.8794 0.8746
21.5398 3200 16.0382 16.4589 0.8742 0.8752 0.8766 0.8723
21.8763 3250 16.0279 16.4668 0.8752 0.8766 0.8773 0.8728
22.2129 3300 16.0327 16.4737 0.8742 0.8768 0.8773 0.8727
22.5494 3350 15.979 16.4686 0.8740 0.8771 0.8771 0.8722
22.8860 3400 15.9622 16.4736 0.8743 0.8760 0.8765 0.8721
23.2225 3450 15.9881 16.4802 0.8743 0.8757 0.8755 0.8723
23.5591 3500 15.9482 16.4821 0.8725 0.8761 0.8761 0.8710
23.8957 3550 15.9228 16.4996 0.8726 0.8748 0.8751 0.8709
24.2322 3600 15.9418 16.4973 0.8709 0.8728 0.8734 0.8699
24.5688 3650 15.896 16.4985 0.8696 0.8716 0.8727 0.8686
24.9053 3700 15.8788 16.5172 0.8691 0.8715 0.8717 0.8662
25.2419 3750 15.9147 16.5062 0.8677 0.8706 0.8712 0.8662
25.5785 3800 15.857 16.5058 0.8683 0.8717 0.8732 0.8663
25.9150 3850 15.8291 16.5207 0.8674 0.8702 0.8706 0.8644
26.2516 3900 15.8802 16.5233 0.8678 0.8697 0.8714 0.8664
26.5881 3950 15.846 16.5170 0.8686 0.8713 0.8717 0.8655
26.9247 4000 15.8012 16.5336 0.8663 0.8682 0.8699 0.8635

Framework Versions

  • Python: 3.10.10
  • Sentence Transformers: 3.0.1
  • Transformers: 4.41.2
  • PyTorch: 2.1.2+cu121
  • Accelerate: 0.33.0
  • Datasets: 2.19.1
  • Tokenizers: 0.19.1

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MatryoshkaLoss

@misc{kusupati2024matryoshka,
    title={Matryoshka Representation Learning}, 
    author={Aditya Kusupati and Gantavya Bhatt and Aniket Rege and Matthew Wallingford and Aditya Sinha and Vivek Ramanujan and William Howard-Snyder and Kaifeng Chen and Sham Kakade and Prateek Jain and Ali Farhadi},
    year={2024},
    eprint={2205.13147},
    archivePrefix={arXiv},
    primaryClass={cs.LG}
}

TripletLoss

@misc{hermans2017defense,
    title={In Defense of the Triplet Loss for Person Re-Identification}, 
    author={Alexander Hermans and Lucas Beyer and Bastian Leibe},
    year={2017},
    eprint={1703.07737},
    archivePrefix={arXiv},
    primaryClass={cs.CV}
}