matt-wisdom/KEmbed-naija-v3

SentenceTransformer based on MattWhatever/KEmbed-naija-v2

This is a sentence-transformers model finetuned from MattWhatever/KEmbed-naija-v2. It maps sentences & paragraphs to a 1024-dimensional dense vector space and can be used for retrieval.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: MattWhatever/KEmbed-naija-v2
• Maximum Sequence Length: 256 tokens
• Output Dimensionality: 1024 dimensions
• Similarity Function: Cosine Similarity
• Supported Modality: Text

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'transformer_task': 'feature-extraction', 'modality_config': {'text': {'method': 'forward', 'method_output_name': 'last_hidden_state'}}, 'module_output_name': 'token_embeddings', 'architecture': 'XLMRobertaModel'})
  (1): Pooling({'embedding_dimension': 1024, 'pooling_mode': 'cls', 'include_prompt': True})
  (2): Normalize({})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("MattWhatever/KEmbed-naija-v3")
# Run inference
sentences = [
    'Nigeria Elections 2019: Ezekwesili ti jẹ́ mínísítà ní Nàìjíríà lẹ́ẹ̀ mejì ọ̀tọ̀ọ̀tọ̀',
    "Ezekwesili ti fi ìgbà kan jẹ́ igbakeji ààrẹ Banki Agbaye éka ilẹ̀ Afrika\n\nIlumọọka ni Ọmọwe Ezekwesili, ẹni ti o ti di oniruuru ipo mu ni Naijiria ati agbaye . \n\nYatọ si pe o jẹ ọkan pataki lara oludasilẹ ẹgbẹ ajafẹtọ ọmọniyan #BBOG to n ja fun ominira awọn ọmọbinrin to wa ni ahamọ ikọ Boko Haram, Ezekwesili jẹ ọkan lara awọn ọmọ Naijiria to n fi ojoojumọ ke tantan lori eto iṣakoso orilẹede naa.\n\nIpinlẹ Anambra la ti bi Oby, ni ọjọ́ kejidinlọgbọn, osu igbe ọdun 1963. O si kẹkọọ gboye gẹgẹ bi akọṣẹmọṣẹ iṣiro owo.\n\nIgba meji ọtoọtọ lo ti ṣe minista ni Naijiria, to si tun ti figba kan jẹ igbakeji aarẹ Banki agbaye, ẹka ilẹ adulawọ.  \n\nỌ̀kan lára olùdásílẹ̀ ẹgbẹ́ #BBOG ni Oby Ezekwesili\n\nMinisita Ohun Alumọni láàrin ọdun 2005 si 2006\n\nNi ọdun 2005 ni Aarẹ Olusegun Obasanjo yan Ọmoọwe Ezekwesili gẹgẹ bi minisita ohun alumọni inu ilẹ, lẹyin to ti jẹ adari ẹka to n ri si amojuto eto isuna , nibi to ti gba inagijẹ 'Mama Due Process'.\n\nGẹgẹ bi minisita ohun alumọni, o ṣe agbatẹru abadofin to ni ṣe pẹlu alumọni ati iwakusa 'Minerals and Mining Act', idasilẹ ileeṣe iwakusa orilẹede Naijiria, o si tun rii daju pe ẹka naa rọrun lati wọ fun awọn aladaani.\n\nMinisita Eto Ẹkọláàrin ọdun 2006 si 2007\n\nIpo minista ohun alumọni inu ilẹ, ni ọmọwe Ezekwesili wa, ti aarẹ nigba naa, Oloye Obasanjo tun ti sọ ọ di minista eto ẹkọ labẹ iṣakoso rẹ̀. \n\nGẹgẹ bi minisita eto ẹkọ, Ezekwesili ṣe atunto ẹka etò ẹkọ orilẹede Naijiria. Oun lo tun ṣe agbatẹru ibaṣepọ laarin ijọba ati aladaani fun eto ẹkọ to yanranti.\n\nEzekwesili ti fi ìgbà kan jẹ́ igbakeji ààrẹ Banki Agbaye éka ilẹ̀ Afrika\n\nIgbakeji-aarẹ Banki Agbaye laarin 2007 si 2012\n\nNi ọdun 2007 ni arabinrin Ezekwesili di igbakeji aarẹ Banki agbaye, ni kete to fi ipo minisita eto ẹkọ silẹ ni Naijiria, ipo to dimu di ọdun 2012.\n\nGẹgẹ bi igbakeji-aarẹ Banki agbaye ẹka ilẹ Adulawọ yii, oun lo jẹ alamojuto gbogbo ohun to n lọ kaakiri awọn orilẹede mejidinlaadọta to wa ni 'Sub Saharan Afirika', o si tun jẹ adari owoyaa to le ni ogoji biliọnu dọla. \n\nEzekwesili sọ pé àsìkò ti tó láti gba agbára lọ́wọ́ ẹgbẹ́ òṣèlú PDP àti APC\n\nAbẹ ẹgbẹ oṣelu ACPN ni o ti n dije sipo aarẹ orilẹede Naijiria\n\nỌmọwe Ezekwesili kede ipinnu rẹ lati dupo aarẹ orilẹede Niajiria ninu ọdun 2018 pẹlu alaye pe awọn ẹgbẹ oṣelu mejeeji to ti ṣe ijọba ri jẹ ọbayejẹ, ti ko si yẹ ki wọn wa ni ijọba mọ lorilẹede eyi.\n\nỌgbẹni Ganiyu Galadima ni wọn yoo jọ maa dije gẹgẹ bi igbakeji. \n\n#BBCNigeria2019",
    'Cheta na akụkọ si n\'aka ndị uweojii kwuru na opekatampe mmadụ anọ anwụyụọla anya ebe ọtụtụ merụrụ ahụ,\n\nn\'ọgụ dapụtara n\'etiti otu nzuzo abụọ n\'ụlọ akwụkwọ \'Federal Polytechnic\' Nekede dị n\'Imo Steeti.\n\nN\'ihi ọgụ na ọgbaghara a, ụmụ akwụkwọ na ndị nkụzi ụlọakwụkwọ ahụ nọzi n\'egwu na ihe ize ndụ ugbu a.\n\nOtu nwata akwụkwọ bụ Frank hụrụ ka ihe si mee gwara BBC Igbo na nkpamkpa ọjọ a malitere n\'izuụka gara aga n\'etiti otu nzuzo abụọ bụ \'Aye\' na \'Bagger\'.Frank kwuru na o bidoro mgbe ahụtara ozu nwata akwụkwọ a, bụkwa onye na-edozi isi n\'ihu ụlọ akwụkwọ ahụ. \n\nNke a kpatara ogbugbu e gburu nwata akwụkwọ ọzọ na-agụ ihe ọmụmụ \'Public Administration\' n\'ogbe ebe akpọrọ South Africa.N\'okwu ya "Nke a mere ha jiri bido gbuwe mmadụ aghara aghara. Ka ọ dị ugbu a, eji m n\'aka na ihe karịrị mmadụ anọ anwụọla, mana ejiri m anya m hụ ozu mmadụ anọ".\n\nỌnụ na-ekwuru ndị uweojii n\'Imo Steeti bụ Orlando Ikeokwu gwara BBC Igbo na ọ bụ eziokwu na mmadụ anọ anwụọla.\n\nO kwuru na ndị uweojii ejidela mmadụ iri ma nata ha egbe ruru ise dịka ha ka na-eme nyocha iji chọpụta otu nzuzo na-akpa arụ a.\n\nAkụkọ ndị ga-amasị gị:',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 1024]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities)
# tensor([[1.0000, 0.6367, 0.3613],
#         [0.6367, 1.0078, 0.4258],
#         [0.3613, 0.4258, 0.9961]], dtype=torch.bfloat16)

Evaluation

Metrics

Information Retrieval

• Dataset: nigerian-val
• Evaluated with InformationRetrievalEvaluator

Metric	Value
cosine_accuracy@1	0.811
cosine_accuracy@3	0.91
cosine_accuracy@5	0.927
cosine_accuracy@10	0.949
cosine_precision@1	0.811
cosine_precision@3	0.3033
cosine_precision@5	0.1854
cosine_precision@10	0.0949
cosine_recall@1	0.811
cosine_recall@3	0.91
cosine_recall@5	0.927
cosine_recall@10	0.949
cosine_ndcg@10	0.8848
cosine_mrr@10	0.8638
cosine_map@100	0.8651

Training Details

Training Dataset

Unnamed Dataset

• Size: 57,090 training samples

• Columns: anchor and positive

• Approximate statistics based on the first 1000 samples:

  	anchor	positive
  type	string	string
  details	min: 7 tokens mean: 24.11 tokens max: 81 tokens	min: 13 tokens mean: 225.08 tokens max: 256 tokens

• Samples:

  anchor	positive
  Where is Cosmas Maduka from?	Maduka, an Igbo Christian from Nnewi, Anambra State was born into the family of Mr. Peter and Mrs. Rose Maduka in the City of Jos. Maduka began his entrepreneurial journey at the age of six, two years after his father died. He dropped out of primary school and started hawking Akara, a popular Nigerian food staple made from beans to support his mother.
  Emma Coronel Aispuro: Matar hamshakin mai safarar miyagun kwayoyi ta yi faduwar bakar tasa	Emma Coronel Aispuro

  Tagogin gidan yarin, Cibiyar Tsare Manya a Alexandria, an yanka su ne ta cikin ginin jan bulo, kuma a nan ne inda ake tsare da Emma Coronel Aispuro ita kadai a wani tsukakken daki.

  A cikin dakin, in ji lauyar ta Mariel Colón Miro, tana karanta litattafan kagaggun labarai, don debe kewa.

  Yanayin gidan yarin na da matukar banbanci da irin rayuwar da ta saba yi a da.

  Watanni kadan da suka gabata, ta yi aniyar kaddamar da wani kamfanin zayyana tufafin kawa da ake kira El Chapo Guzman. (Ma'auratan na da wata alama da ken una matsayinsu a kasar Mexico, kana ita ma 'yar su ta yi fice a fannin ado da kayan kawa ta hanyar amfani da sunan shi.

  A lokacin da na tattauna da ita a birnin New York yayin da ake yi wa mijinta shari'a a shekarar 2019, tana sanye da sarkoki da agogo mafi tsada.

  Amma a farkon wannan shekarar, an cafke Coronel mai shekkaru 31 a filin saukar jiragen saman a kasa da kas ana jihar Virginia kuma aka tuhume ta da taimaka wa mijinta, hamshakin mai f... | | IGP enyela iwu ka echesie ụlọomeiwu ukwu ike | Ike Ekweremadu nọchiri Bukola Saraki na nzukọ ebe a bịara buru ọfọ a

  Onyeisi ndị uweojii bụ Idris Ibrahim, nyekwara iwu ka ndị uweojii tinyekwuo uchu na nchekwa ụlọomeiwu dịka ozi a na-apụta.

  Isi ụlọọrụ ndị uweojii kwuru nke a n'akwụkwọ ozi nke osote ọnụ na-ekwuchitere ndị uweojii Naijiria bụ SP Aremu Adeniran, tinyere aka sị na a hụrụ ọfọ ahụ n'okpuru akwa ụgbọala nke a kpọrọ 'Flyover' na bekee dị na City Gate, n'Abụja ebe otu nwa amadị hụrụ ya ma kpokuo ndị uweojii.

  N'ụbọchị Wenesde, ndị Sineti nyere ndị uweojii na ndị nchekwa ndị ọzọ otu ụbọchị ka ha chọta ya bụ ọfọ na-efu.

  Ozi ahụ kwuru si: "Onyeisi ndị uweojii bụ Ibrahim Idris mere ngwangwa hiwe otu ọkpọka na nyocha ma nye ha iwu ka ha gbachie isiobodo Naijiria mee ezi akwụsị-enyochaa iji nwụchị ndị omekome ahụ ma nata ha ọfọ.

  "Ndị uweojii wakpokwara ebe ndị omekome na-anọkarị nke mere ka ndị omekome ahụ hapụ ọfọ ahụ n'okpuru akwa ụgbọala dị na City Gate gbalaga, ebe otu nwa amadi hụrụ ala nna anyị n'anya hụrụ ya ma bekuo nd... |

• Loss: MatryoshkaLoss with these parameters:

  {
      "loss": "MultipleNegativesRankingLoss",
      "matryoshka_dims": [
          1024,
          512,
          256,
          128,
          64
      ],
      "matryoshka_weights": [
          1,
          1,
          1,
          1,
          1
      ],
      "n_dims_per_step": -1
  }
  

Evaluation Dataset

Unnamed Dataset

• Size: 7,128 evaluation samples

• Columns: anchor and positive

• Approximate statistics based on the first 1000 samples:

  	anchor	positive
  type	string	string
  details	min: 7 tokens mean: 24.89 tokens max: 74 tokens	min: 9 tokens mean: 223.88 tokens max: 256 tokens

• Samples:

  anchor	positive
  Akụkọ kachasị n'abalị a: Ụlọ omeiwu abụọ agbakụtala Ibrahim idris azụ	Ndị omeiwu na onyeisi ndị uweojii na-akwata ya.

  Ndị omeiwu ukwu na nke nta ekwuola na ha enweghi ntụkwasị obi ebe onyeisi ndị uweojii bụ Ibrahim Idris nọ.

  Nke a sotere nzụkọ mberede pụrụiche nke ụlọomeiwu abụọ ahụ mere n'ime ụlọ.

  Naịjirịa

  Buhari na ndị ọchịagha mere ọgbakọ taa

  Onyeisiala Muhammadu Buhari na ndị ọchị agha niile na Naịjirịa nwere ọgbakọ taa ebe ha kpara etu nchekwa ala Naịjirịa dị ugbua.

  Nke a na-esote ọgbakọ onyeisiala na osote ya bụ Yemi Osibanjo mere ụnyaahụ n'etiti ndị ọchị agha ahụ n'oge dị iche iche .

  Ebe ọzọ na Naịjirịa

  Naịjirịa dị nkwadebe ịnabata ndị ọbịa.

  Onyeisiala Buhari ekwola na Naịjirịa nwere nchekwa ịnabata ndị ọbịa na-abịa ilegharị anya nke a kpọrọ tourism na Bekee.

  Buhari kwuru nke a oge onyeisi na-ahụ maka njegharị n'otu mbaụwa United Nations bú Zurab Pololikashvili gara leta ya.

  Akụkọ mba ofesi

  Agụiyi dọgburu mmadụ na Etiopia

  Agụiyi na-ebi na ezu ụfọdụ

  Agụiyi adọgbuola ụkọchukwu na-eme ndị ụka ya mmirichukwu n'akụkụ ezu na mba Etiopịa.

  ... | | Donald Trump coronavirus: Meet "Hope Hicks" White House Communications Director wey catch Covid 19 | Di president adviser wey be 31-year-old na former model wey no dey like to keep low profile. She no too like publicity.

  Hope Hicks replace Anthony Scaramucci as di president communications director when di presido sack am afta 10 days for office in 2017.

  She no get any background for politics, but she dey connected to di Trump family since five years ago.

  Her political career wit Oga Trump get as e be - as she dey resign from one role enta another she resigned from one role, only to return later to another different position for im team.

  So how pesin wit that kain low profile take get one of di most important jobs for di US goment?

  She sabi Donald through Ivanka Trump

  Hope Hicks start her career for public relations, and Ivanka Trump fashion company na one of her clients.

  She bin model for Ralph Lauren - and she also appear for di cover of a Gossip Girl spin-off book - she model some of Ivanka clothes as part of her job.

  Na as she dey work wit Oga Trump eldest daughter mean sa... | | Americans in Africa: Wo ìdí tí àwọn ọmọ ilẹ̀ Amẹ́ríkà kan ṣe ń kó bọ̀ wá fi ilẹ̀ Áfríkà sebùgbé | Lara irufẹ awọn bẹẹ ni Brian, Rukiya ati Marcus ti wọn fi Amẹrika silẹ lọ si Namibia, Tanzania ati Uganda lati maa gbe.

  Awọn eeyan naa sọ fun BBC pe eredi ti wọn ṣe fi Amẹrika silẹ ko ṣẹyin bi iwa ẹlẹyamẹya ṣe di tọrọfọnkale nibẹ.

  Marcus Pace sọ pe oun ti maa n ṣabẹwo si Ugandan loorekoore lati ọjọ to ti pẹ, ṣugbọn oun ti wa pinnu lati fidi mọlẹ si Afrika bayii.

  O ni o rọrun lati gbe nilẹ adulawọ ju ati gbe ni Amẹrika lọ nitori iṣe ni oun maa n ṣe ni gbogbo igba latari awọn owo ori gọbọi atawọn nnkan miran.

  Marcus ṣalaye pe ọkan oun balẹ nitori ko si ẹni to n gbiyan lati yin oun nibọn nitori pe oun jẹ alawọ dudu.

  Bakan naa ni arabinrin Rukiya McNair ni inu awọn ọmọ oun dun pupọ lati gbe nilẹ adulawọ.

  Ẹ̀yin obìnrin tẹ́ẹ fẹ́ wá bímọ ní America, ọ̀nà ti tì pa látònì! - Trump

  Bí Amẹrika bá dínà mọ́ Nàíjíríà láti wá sílẹ̀ wọn, ewu ń bẹ fún wa - Lai Muhammed

  Ìtàn Olaniyi Balogun, ọ̀jọ̀gbọ́n tó fi Nàìjíríà sílẹ̀ lọ gba iṣẹ̀ àgbẹ̀ l'Ámẹ́ríkà

  O ṣalaye awọn oun ti oju rẹ ri nigba to n d... |

• Loss: MatryoshkaLoss with these parameters:

  {
      "loss": "MultipleNegativesRankingLoss",
      "matryoshka_dims": [
          1024,
          512,
          256,
          128,
          64
      ],
      "matryoshka_weights": [
          1,
          1,
          1,
          1,
          1
      ],
      "n_dims_per_step": -1
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• eval_strategy: steps
• gradient_accumulation_steps: 16
• learning_rate: 5e-06
• num_train_epochs: 1
• lr_scheduler_type: cosine
• warmup_steps: 0.05
• bf16: True
• dataloader_num_workers: 2
• load_best_model_at_end: True

All Hyperparameters

Click to expand

• do_predict: False
• eval_strategy: steps
• prediction_loss_only: True
• per_device_train_batch_size: 8
• per_device_eval_batch_size: 8
• gradient_accumulation_steps: 16
• eval_accumulation_steps: None
• torch_empty_cache_steps: None
• learning_rate: 5e-06
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1.0
• num_train_epochs: 1
• max_steps: -1
• lr_scheduler_type: cosine
• lr_scheduler_kwargs: None
• warmup_ratio: None
• warmup_steps: 0.05
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• enable_jit_checkpoint: False
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• use_cpu: False
• seed: 42
• data_seed: None
• bf16: True
• fp16: False
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• local_rank: -1
• ddp_backend: None
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 2
• dataloader_prefetch_factor: None
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: True
• ignore_data_skip: False
• fsdp: []
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• parallelism_config: None
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch_fused
• optim_args: None
• group_by_length: False
• length_column_name: length
• project: huggingface
• trackio_space_id: trackio
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: None
• hub_always_push: False
• hub_revision: None
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_for_metrics: []
• eval_do_concat_batches: True
• auto_find_batch_size: False
• full_determinism: False
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• include_num_input_tokens_seen: no
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: False
• use_liger_kernel: False
• liger_kernel_config: None
• eval_use_gather_object: False
• average_tokens_across_devices: True
• use_cache: False
• prompts: None
• batch_sampler: batch_sampler
• multi_dataset_batch_sampler: proportional
• router_mapping: {}
• learning_rate_mapping: {}

Training Logs

Epoch	Step	Training Loss	Validation Loss	nigerian-val_cosine_ndcg@10
0.1121	25	2.2809	-	-
0.2242	50	1.7451	-	-
0.3362	75	1.4996	-	-
0.4483	100	1.4651	-	-
0.5335	119	-	1.0193	0.8848
0.5604	125	1.3706	-	-
0.6725	150	1.2904	-	-
0.7845	175	1.3165	-	-
0.8966	200	1.3392	-	-

• The bold row denotes the saved checkpoint.

Training Time

• Training: 3.6 hours

Framework Versions

• Python: 3.12.13
• Sentence Transformers: 5.4.0
• Transformers: 5.0.0
• PyTorch: 2.10.0+cu128
• Accelerate: 1.13.0
• Datasets: 4.8.5
• Tokenizers: 0.22.2

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MatryoshkaLoss

@misc{kusupati2024matryoshka,
    title={Matryoshka Representation Learning},
    author={Aditya Kusupati and Gantavya Bhatt and Aniket Rege and Matthew Wallingford and Aditya Sinha and Vivek Ramanujan and William Howard-Snyder and Kaifeng Chen and Sham Kakade and Prateek Jain and Ali Farhadi},
    year={2024},
    eprint={2205.13147},
    archivePrefix={arXiv},
    primaryClass={cs.LG}
}

MultipleNegativesRankingLoss

@misc{oord2019representationlearningcontrastivepredictive,
      title={Representation Learning with Contrastive Predictive Coding},
      author={Aaron van den Oord and Yazhe Li and Oriol Vinyals},
      year={2019},
      eprint={1807.03748},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/1807.03748},
}