matt-wisdom/KEmbed-naija-v1

SentenceTransformer based on BAAI/bge-m3

This is a sentence-transformers model finetuned from BAAI/bge-m3. It maps sentences & paragraphs to a 1024-dimensional dense vector space and can be used for retrieval.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: BAAI/bge-m3
• Maximum Sequence Length: 256 tokens
• Output Dimensionality: 1024 dimensions
• Similarity Function: Cosine Similarity
• Supported Modality: Text

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'transformer_task': 'feature-extraction', 'modality_config': {'text': {'method': 'forward', 'method_output_name': 'last_hidden_state'}}, 'module_output_name': 'token_embeddings', 'architecture': 'XLMRobertaModel'})
  (1): Pooling({'embedding_dimension': 1024, 'pooling_mode': 'cls', 'include_prompt': True})
  (2): Normalize({})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("sentence_transformers_model_id")
# Run inference
sentences = [
    'What position did Bukola Saraki hold in the Nigerian Senate?',
    "Sinima to gun ni ọrọ oṣelu Naijiria, ko fẹ ẹ ni opin: Nibi ti ọrọ de e duro yii, ko si ẹni to le sọ pe ibi kan ni yoo ja si nigbẹyin. Nigba ti awuyewuye yii bẹrẹ, Adams Oshiomole lo dipo alaga ẹgbẹ APC mu. Ni ba ti se n sọrọ yi, ile ẹjọ giga ti yẹ aga nidi Oshiomole ti awọn miran si ti n ja witiwiti lati wa ni ipo alaga. Labẹ pe ibẹrẹ kọ l'onisẹ, afi ẹni ba laa ja, awọn ti o fi ẹgbẹ oselu APC silẹ fun Oshiomole bi Aarẹ ile asofin agba tẹlẹ ri Bukola Saraki ni wọn wa n gba Godwin Obaseki lalejo ninu ẹgbẹ oselu PDP bayi. Njẹ ajọsepọ awọn to kora jọ lati doju ija kọ ara wọn wọnyi, yoo jẹ ọlọjọ pipẹ tabi awọn naa yoo pada wa tutọ si ara wọn loju? Oju re e, iran ree lọrọ to wa nilẹ yi.",
    'Ọjọ Aje, Mọnde, ọsẹ yii, nijọba ipinlẹ Kwara, latọwọ Kọmisanna to n ri sọrọ ijọba ibilẹ ati lọba lọba, Aliyu Saifudeen, buwọ lu Ismail Yahaya Alebiosu gẹgẹ bii Olupo ti ilu Ajasẹ-Ipo tuntun. Kọmiṣanna sọ pe, awọn yan Ismall Yahaya Alebiosu gẹgẹ bii Olupo, nitori oun lo tọ si ipo naa. O fi kun un pe gomina fọwọ si iyansipo rẹ, ti yoo si maa ṣe atilẹyin fun un nigbakugba ti ọba alaye naa ba nilo iranlọwọ ijọba. Nigba ti Ọba Alebiosu n tẹwọ gba iwe iyansipo rẹ, o dupẹ lọwọ Gomina ipinlẹ Kwara, Abdulrahman Abdulrasaq, fun bo ṣe tẹle ohun ti awọn araalu n fẹ, to si buwọ lu iyansipo oun, o waa jẹjẹẹ pe oun ko ni i kuna lati maa ṣe ojuse oun, bẹẹ lo ṣeleri pe oun ko ni i ja awọn eeyan ilu naa kulẹ nigba kankan.',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 1024]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities)
# tensor([[1.0000, 0.4805, 0.3301],
#         [0.4805, 1.0000, 0.5117],
#         [0.3301, 0.5117, 0.9961]], dtype=torch.bfloat16)

Evaluation

Metrics

Triplet

• Dataset: nigerian-val
• Evaluated with TripletEvaluator

Metric	Value
cosine_accuracy	0.7925

Training Details

Training Dataset

Unnamed Dataset

• Size: 4,466 training samples
• Columns: anchor and positive
• Approximate statistics based on the first 1000 samples:

  	anchor	positive
  type	string	string
  details	min: 6 tokens mean: 16.51 tokens max: 79 tokens	min: 13 tokens mean: 141.29 tokens max: 256 tokens

• Samples:

  anchor	positive
  Where is Cosmas Maduka from?	Maduka, an Igbo Christian from Nnewi, Anambra State was born into the family of Mr. Peter and Mrs. Rose Maduka in the City of Jos. Maduka began his entrepreneurial journey at the age of six, two years after his father died. He dropped out of primary school and started hawking Akara, a popular Nigerian food staple made from beans to support his mother.
  Which year was Mayotte created?	Mayotte broke away from the Comoros and remained with France after the Comoros declared its independence following in the 1974 referendum. Mayotte became an overseas department on 31 March 2011 and became an outermost region of the European Union on 1 January 2014, following a March 2009 referendum with an overwhelming result in favour of the department status.
  Ọdun wo ni Deborah Oluwaseyi Joshua gbe awo orin rẹ akọkọ jade?	In July 2015, she signed a record deal with Island Records. In 2015, she signed a two-year endorsement deal with Pepsi. Shay released her debut studio album Seyi or Shay in November 2015.

• Loss: MatryoshkaLoss with these parameters:

  {
      "loss": "MultipleNegativesRankingLoss",
      "matryoshka_dims": [
          1024,
          512,
          256,
          128,
          64
      ],
      "matryoshka_weights": [
          1,
          1,
          1,
          1,
          1
      ],
      "n_dims_per_step": -1
  }
  

Evaluation Dataset

Unnamed Dataset

• Size: 567 evaluation samples
• Columns: anchor and positive
• Approximate statistics based on the first 567 samples:

  	anchor	positive
  type	string	string
  details	min: 6 tokens mean: 16.5 tokens max: 51 tokens	min: 10 tokens mean: 138.19 tokens max: 256 tokens

• Samples:

  anchor	positive
  A cikin wace jami'a ce Manouchka Kelly Labouba ta kammala digirinta na PhD?	In 2005, she studied Film & Media Studies at University of California, Santa Barbara. Then, in 2007, he obtained Master de Recherche, Arts from Bordeaux Montaigne University. Later he graduated with a M.A. Critical Studies and then Ph.D. Cinema and Media Studies from University of Southern California to obtain and also graduated with a Certificate in the Business of Entertainment.
  Who represented Kaduna South in the 2019 Nigerian Senate?	The 2019 Nigerian Senate election in Kaduna State held on February 23, 2019, to elect members of the Nigerian Senate to represent Kaduna State. Kwari Suleiman Abdu representing Kaduna North, Sani Uba representing Kaduna Central both won on the platform of All Progressives Congress. while Laah Danjuma Tella representing Kaduna South returned to parliament on the platform of People's Democratic Party Results.
  Who is the former governor of Lagos state that also ran for Nigerian President?	Aare Muhammadu Buhari ti ki Asiwaju Bola Ahmed Tinubu u oriire, bee lo si ke si gbogbo awon omo Naijiria ki won fowosowopo pelu aare adiboyan naa. Buhari ni, ki awon ti ibo naa ko dun mo ninu gba ile ejo lo, ki won loo rojo nibe, nitori ohun to sele ninu ibo yii fihan pe ko si mago-mago kankan. O ni, Tinubu funra re padanu ipinle Eko, tii se ipinle re, bee loun naa padanu ipinle oun, tii se ipinle Katsina, eyi si fihan pe ko si eru kan ninu eto idibo naa, eni tawon eeyan fe lo wole ibo. O loun setan ati seto igbejoba pada fun Tinubu atawon eeyan re ko too di pe ojo igbejobo sile de ninu osu karun-un, odun yii.

• Loss: MatryoshkaLoss with these parameters:

  {
      "loss": "MultipleNegativesRankingLoss",
      "matryoshka_dims": [
          1024,
          512,
          256,
          128,
          64
      ],
      "matryoshka_weights": [
          1,
          1,
          1,
          1,
          1
      ],
      "n_dims_per_step": -1
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• eval_strategy: steps
• gradient_accumulation_steps: 16
• learning_rate: 5e-06
• num_train_epochs: 1
• lr_scheduler_type: cosine
• warmup_steps: 0.05
• bf16: True
• dataloader_num_workers: 2
• load_best_model_at_end: True

All Hyperparameters

Click to expand

• do_predict: False
• eval_strategy: steps
• prediction_loss_only: True
• per_device_train_batch_size: 8
• per_device_eval_batch_size: 8
• gradient_accumulation_steps: 16
• eval_accumulation_steps: None
• torch_empty_cache_steps: None
• learning_rate: 5e-06
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1.0
• num_train_epochs: 1
• max_steps: -1
• lr_scheduler_type: cosine
• lr_scheduler_kwargs: None
• warmup_ratio: None
• warmup_steps: 0.05
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• enable_jit_checkpoint: False
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• use_cpu: False
• seed: 42
• data_seed: None
• bf16: True
• fp16: False
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• local_rank: -1
• ddp_backend: None
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 2
• dataloader_prefetch_factor: None
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: True
• ignore_data_skip: False
• fsdp: []
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• parallelism_config: None
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch_fused
• optim_args: None
• group_by_length: False
• length_column_name: length
• project: huggingface
• trackio_space_id: trackio
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: None
• hub_always_push: False
• hub_revision: None
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_for_metrics: []
• eval_do_concat_batches: True
• auto_find_batch_size: False
• full_determinism: False
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• include_num_input_tokens_seen: no
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: False
• use_liger_kernel: False
• liger_kernel_config: None
• eval_use_gather_object: False
• average_tokens_across_devices: True
• use_cache: False
• prompts: None
• batch_sampler: batch_sampler
• multi_dataset_batch_sampler: proportional
• router_mapping: {}
• learning_rate_mapping: {}

Training Logs

Epoch	Step	Validation Loss	nigerian-val_cosine_accuracy
0.7033	2	8.0186	0.8019
1.3516	4	5.7152	0.7925
2.0	6	5.6023	0.7925
0.9714	17	1.8886	0.7925

• The bold row denotes the saved checkpoint.

Training Time

• Training: 16.3 minutes

Framework Versions

• Python: 3.12.13
• Sentence Transformers: 5.4.0
• Transformers: 5.0.0
• PyTorch: 2.10.0+cu128
• Accelerate: 1.13.0
• Datasets: 4.8.5
• Tokenizers: 0.22.2

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MatryoshkaLoss

@misc{kusupati2024matryoshka,
    title={Matryoshka Representation Learning},
    author={Aditya Kusupati and Gantavya Bhatt and Aniket Rege and Matthew Wallingford and Aditya Sinha and Vivek Ramanujan and William Howard-Snyder and Kaifeng Chen and Sham Kakade and Prateek Jain and Ali Farhadi},
    year={2024},
    eprint={2205.13147},
    archivePrefix={arXiv},
    primaryClass={cs.LG}
}

MultipleNegativesRankingLoss

@misc{oord2019representationlearningcontrastivepredictive,
      title={Representation Learning with Contrastive Predictive Coding},
      author={Aaron van den Oord and Yazhe Li and Oriol Vinyals},
      year={2019},
      eprint={1807.03748},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/1807.03748},
}