harshdevjha/indicclaimver-labse-retriever

IndicClaimVer LaBSE Retriever

A LaBSE-based multilingual sentence retriever fine-tuned for claim → evidence retrieval, built for the IndicClaimVer@FIRE-2026 shared task on claim verification across English, Hindi, Bengali, and CodeMix.

Why This Model Exists

The shared task's Subtask 1 (closed-evidence claim verification) requires, given a claim, retrieving the top-3 most relevant pieces of evidence from a large fixed pool (13,333 items at dev time, 28,618 at test time) spanning four languages/scripts. No baseline retriever was provided by the organizers, so this model was built from scratch on top of a pretrained multilingual sentence encoder.

Off-the-shelf sentence-transformers/LaBSE is a strong general-purpose multilingual encoder, but it isn't specialized for the specific way claims and evidence are phrased in this task — claims are often short, headline-style statements (sometimes in CodeMix or a mix of scripts), while evidence passages are longer news/fact-check style paragraphs. Fine-tuning on labeled (claim, gold evidence) pairs from the task's training set closes that gap, teaching the model to pull the two closer together in embedding space specifically for this claim-to-evidence retrieval pattern, rather than relying on generic semantic similarity alone.

This retriever is one half of a two-model pipeline (paired with a fine-tuned mDeBERTa veracity classifier) that powers both subtasks of IndicClaimVer@FIRE-2026 end-to-end — see the full pipeline repo for how it fits together with FAISS indexing, the classifier, and the open-web evidence pipeline used in Subtask 2.

What Changed From the Base Model

Fine-tuning was done with MultipleNegativesRankingLoss on 4,536 (claim, gold evidence) training pairs, treating each claim's true evidence as the positive and other evidence in the batch as in-batch negatives — a standard, efficient way to specialize a sentence encoder for retrieval without needing explicitly mined hard negatives.

The result, measured on the task's dev set (1,134 claims against a 13,333-item evidence pool):

Metric	Off-the-shelf LaBSE	This fine-tuned model
Success@3	0.664	0.793
nDCG@3	0.464	0.502

Success@3 — whether at least one gold evidence item lands in the top-3 retrieved — improved by nearly 13 points, which directly matters here since Subtask 1's downstream veracity classifier only ever sees the top-3 retrieved evidence, not the full pool.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: sentence-transformers/LaBSE
• Maximum Sequence Length: 256 tokens
• Output Dimensionality: 768 dimensions
• Similarity Function: Cosine Similarity
• Supported Modality: Text
• Languages: English, Hindi, Bengali, CodeMix (Hindi/Bengali-English mixed script)
• Fine-tuning task: Claim → evidence retrieval for the IndicClaimVer@FIRE-2026 shared task

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face
• Full pipeline (retriever + classifier + open-web verification): IndicClaim-FIRE on GitHub
• Paired classifier: indicclaimver-mdeberta-classifier

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'transformer_task': 'feature-extraction', 'modality_config': {'text': {'method': 'forward', 'method_output_name': 'last_hidden_state'}}, 'module_output_name': 'token_embeddings', 'architecture': 'BertModel'})
  (1): Pooling({'embedding_dimension': 768, 'pooling_mode': 'cls', 'include_prompt': True})
  (2): Dense({'in_features': 768, 'out_features': 768, 'bias': True, 'activation_function': 'torch.nn.modules.activation.Tanh', 'module_input_name': 'sentence_embedding', 'module_output_name': 'sentence_embedding'})
  (3): Normalize({})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("harshdevjha/indicclaimver-labse-retriever")
# Run inference
sentences = [
    'Tarun Tejpal case: Law should not intervene, Says Meira Kumar',
    'Lok Sabha Speaker Meira Kumar on Saturday said the “law should take its own course”, reacting to the alleged sexual assault by Tarun Tejpal, Editor-in-Chief of Tehelka magazine. “Anything where the dignity and security of women is compromised is something of great concern and it should not happen… I think law should take its own course and whatever needs to be done according to law must be done expeditiously,” Kumar told mediapersons here. She was addressing an interactive session on ‘Empowerment of the people and rising violence in society – Contradictory signal for Indian Democracy’ organised by the MCC Chamber of Commerce & Industry.',
    'भारत में कोरोनावायरस के संक्रमण की रफ्तार कम होने का नाम नहीं ले रही है। http://www.covid19india.org के मुताबिक, मंगलवार रात 11:30 बजे तक देश में कोरोनावायरस संक्रमितों की कुल संख्या 11 लाख 92 हजार 151 हो चुकी है। मंगलवार को 37238 नए केस सामने आए, जबकि 670 लोगों की मौत हुई। कोरोना के कारण अब तक देश में 28769 लोगों की जान जा चुकी है। देश में अब एक्टिव केसों की संख्या 4 लाख 10 हजार 665 है, जबकि 7 लाख 52 हजार 312 लोग कोरोना को मात दे चुके हैं। यह लगातार छठा दिन रहा जब देश में कोरोना संक्रमितों की संख्या एक दिन में 35 हजार पार की।',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 768]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities)
# tensor([[ 1.0000,  0.6509,  0.0532],
#         [ 0.6509,  1.0000, -0.0342],
#         [ 0.0532, -0.0342,  1.0000]])

Usage in the IndicClaimVer Pipeline

Within the full pipeline, this model is used to build a FAISS index over an evidence pool, then retrieve the top-3 evidence candidates for each claim at inference time, which are then passed to the paired mDeBERTa classifier for SUPPORTS/REFUTES prediction. See subtask1/build_faiss_index.py and subtask1/retriever.py in that repo for the exact retrieval code this model was trained to work with.

Training Details

Training Dataset

IndicClaimVer@FIRE-2026 Subtask 1 Training Pairs

• Size: 4,536 training samples
• Source: train_subtask1.json from the IndicClaimVer@FIRE-2026 organizers, reduced to (claim, gold evidence) pairs
• Columns: sentence_0 and sentence_1
• Approximate statistics based on the first 100 samples:

  	sentence_0	sentence_1
  type	string	string
  modality	text	text
  details	min: 13 tokens mean: 25.76 tokens max: 43 tokens	min: 61 tokens mean: 153.09 tokens max: 256 tokens

• Samples:

  sentence_0	sentence_1
  মোদির দুনিয়ায় সত্য়কে চাপা যায়, বাস্তবে নয়; কটাক্ষ রাহুলের - Rahul Gandhi	মোদিজির দুনিয়ায় সত্যকে উড়িয়ে দেওয়া যায়, কিন্তু বাস্তবে সত্যকে নির্মূল করা যায় না ।" এ দিন সংসদ চত্বরে সাংবাদিকদের প্রশ্নের জবাবে রাহুল গান্ধি আরও বলেন, "আমাকে যা বলার ছিল, আমি বলেছি এবং এটাই সত্য । তারা যত খুশি উড়িয়ে দিতে পারে, কিন্তু সত্যের জয় হবে ।উল্লেখ্য, 2014 ও 2019 সালে কোনও রাজনৈতিক দলই প্রধান বিরোধী দলের তকমা পায়নি ৷ ফলে গত দশ বছরে লোকসভায় বিরোধী দলনেতা বলে কেউ ছিলেন না ৷ বিরোধীদের মধ্যে সবচেয়ে বেশি সাংসদ থাকায় কংগ্রেসের লোকসভার দলনেতাই কার্যত বিরোধী দলনেতার কাজ করতেন ৷
  जयपुर में पीएम मोदी ने कहा, 'कांग्रेस एक ‘बेल गाड़ी’ है क्योंकि इसके ज्यादातर नेता जमानत पर हैं'	प्रधानमंत्री नरेंद्र मोदी ने जयपुर में एक रैली के दौरान कांग्रेस पर निशाना साधते हुए कहा कि यह पार्टी एक बेल गाड़ी की तरह है क्योंकि इसके अधिकांश नेता जमानत पर बाहर हैं। उन्होंने आरोप लगाया कि कांग्रेस के नेताओं पर भ्रष्टाचार और घोटालों के गंभीर आरोप हैं, जिसके कारण उन्हें अदालतों से जमानत लेनी पड़ती है। मोदी ने इस बयान के जरिए कांग्रेस की विश्वसनीयता पर सवाल उठाया और जनता से भाजपा को समर्थन देने का आग्रह किया। यह टिप्पणी चुनावी माहौल में राजनीतिक प्रतिद्वंद्विता को और तीखा करने वाली मानी जा रही है।
  মুখ্যমন্ত্রীকে শো-কজ রাজ্যপালের, পালটা কড়া সমালোচনা রাজ্য সরকারের - show cause notice issued to CM	মুখ্যমন্ত্রীকে শো-কজ করেছিলেন রাজ্যপাল ৷ তার পালটা মন্ত্রিসভার বৈঠক ডেকে রাজ্যপালের ভূমিকার তীব্র সমালোচনা করল রাজ্য সরকার ৷ পাশাপাশি শো-কজ নোটিশ প্রত্যাহারের কথাও বলা হয়েছে ৷কর্ণাটকের রাজ্যপাল থাওয়ারচাঁদ গেহলত মুখ্যমন্ত্রী সিদ্দারামাইয়াকে একটি বিষয়ে কারণ দর্শানোর নোটিশ দিয়েছিলেন ৷ তা প্রত্যাহার করার পরামর্শ দিয়েছে সরকার ৷ রাজ্যের সাংবিধানিক প্রধান মইসুরু আরবান ডেভেলপমেন্ট অথরিটি (MUDA) সংক্রান্ত একটি অভিযোগের প্রেক্ষিতে বিচারপ্রক্রিয়ায় অনুমোদন দেওয়া হবে না কেন তা জানতে চান প্রশাসনিক প্রধানের কাছে।এই ইস্যুতে মুখ্যমন্ত্রী সিদ্দারামাইয়াকে রাজ্যপালের নোটিশ দেওয়া নিয়ে মন্ত্রিসভার বৈঠকে বৃহস্পতিবার দীর্ঘ আলোচনাও হয়

• Loss: MultipleNegativesRankingLoss with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "cos_sim",
      "gather_across_devices": false,
      "directions": [
          "query_to_doc"
      ],
      "partition_mode": "joint",
      "hardness_mode": null,
      "hardness_strength": 0.0
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• per_device_train_batch_size: 4
• per_device_eval_batch_size: 4
• multi_dataset_batch_sampler: round_robin

All Hyperparameters

Click to expand

• per_device_train_batch_size: 4
• num_train_epochs: 3
• max_steps: -1
• learning_rate: 5e-05
• lr_scheduler_type: linear
• lr_scheduler_kwargs: None
• warmup_steps: 0
• optim: adamw_torch_fused
• optim_args: None
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• optim_target_modules: None
• gradient_accumulation_steps: 1
• average_tokens_across_devices: True
• max_grad_norm: 1
• label_smoothing_factor: 0.0
• bf16: False
• fp16: False
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• use_liger_kernel: False
• liger_kernel_config: None
• use_cache: False
• neftune_noise_alpha: None
• torch_empty_cache_steps: None
• auto_find_batch_size: False
• log_on_each_node: True
• logging_nan_inf_filter: True
• include_num_input_tokens_seen: no
• log_level: passive
• log_level_replica: warning
• disable_tqdm: False
• project: huggingface
• trackio_space_id: None
• trackio_bucket_id: None
• trackio_static_space_id: None
• per_device_eval_batch_size: 4
• prediction_loss_only: True
• eval_on_start: False
• eval_do_concat_batches: True
• eval_use_gather_object: False
• eval_accumulation_steps: None
• include_for_metrics: []
• batch_eval_metrics: False
• save_only_model: False
• save_on_each_node: False
• enable_jit_checkpoint: False
• push_to_hub: False
• hub_private_repo: None
• hub_model_id: None
• hub_strategy: every_save
• hub_always_push: False
• hub_revision: None
• load_best_model_at_end: False
• ignore_data_skip: False
• restore_callback_states_from_checkpoint: False
• full_determinism: False
• seed: 42
• data_seed: None
• use_cpu: False
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• parallelism_config: None
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• dataloader_prefetch_factor: None
• remove_unused_columns: True
• label_names: None
• train_sampling_strategy: random
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• ddp_static_graph: None
• ddp_backend: None
• ddp_timeout: 1800
• fsdp: None
• fsdp_config: None
• deepspeed: None
• debug: []
• skip_memory_metrics: True
• do_predict: False
• resume_from_checkpoint: None
• warmup_ratio: None
• local_rank: -1
• prompts: None
• batch_sampler: batch_sampler
• multi_dataset_batch_sampler: round_robin
• router_mapping: {}
• learning_rate_mapping: {}

Training Logs

Epoch	Step	Training Loss
0.4409	500	0.0173
0.8818	1000	0.0120
1.3228	1500	0.0072
1.7637	2000	0.0063
2.2046	2500	0.0030
2.6455	3000	0.0057

Training Time

• Training: 1.8 hours

Framework Versions

• Python: 3.10.20
• Sentence Transformers: 5.6.0
• Transformers: 5.14.1
• PyTorch: 2.11.0+cu128
• Accelerate: 1.14.0
• Datasets: 5.0.0
• Tokenizers: 0.22.2

Evaluation

Evaluated on dev_subtask1.json (1,134 claims) against the dev-time evidence pool (13,333 items):

Metric	Off-the-shelf LaBSE	Fine-tuned LaBSE (this model)
Success@3	0.664	0.793
nDCG@3	0.464	0.502

Citation

If you use this model, please cite the IndicClaimVer@FIRE-2026 shared task and link back to the pipeline repository and this model card.

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MultipleNegativesRankingLoss

@misc{oord2019representationlearningcontrastivepredictive,
      title={Representation Learning with Contrastive Predictive Coding},
      author={Aaron van den Oord and Yazhe Li and Oriol Vinyals},
      year={2019},
      eprint={1807.03748},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/1807.03748},
}