simcse-bert-base-snli-sup

Supervised SimCSE (Gao, Yao & Chen, EMNLP 2021, Eq. 5: entailment pairs as positives, contradictions as hard negatives) on top of bert-base-uncased, trained on a 100k-record SNLI subset as a course assignment (U2T02, Trends in Data Science). It maps a sentence to a 768-dimensional, L2-normalized vector; compare vectors with cosine similarity.

Team: Julio Cesar De Aquino Castellanos, Gustavo Alexander Fuentes Marin, Valeria Nicol Hernandez Leon, Ricardo Daniel Horta Sanchez, Jose Angel Pech Xool, Lorena Danae Perez Lopez.

This is a reproduction under a reduced budget, not the paper's released checkpoint: it is 5.5 Spearman points below the paper on STS-B test (see Limitations). It is our best model, and it beats SBERT-2019 on STS-B dev and test while training on about 1/30 of its data. The unsupervised companion model is pyrawn/simcse-bert-base-snli-unsup (68.33 test).

Usage

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("pyrawn/simcse-bert-base-snli-sup")
emb = model.encode(["A man is playing a guitar.", "Someone plays an instrument."])
print(model.similarity(emb[0:1], emb[1:2]))  # cosine similarity

Architecture: Transformer (BERT-base, max 512 tokens) → Pooling(cls) (last-layer [CLS] hidden state; this model was trained without an MLP head, and BERT's pretrained pooler is not used) → Normalize.

Training data

Not the paper's data. The paper trains supervised SimCSE on 314k SNLI + MNLI (premise, entailment, contradiction) triplets; this model was trained on 33,351 (premise, entailment) pairs from a fixed 100k-record subset of the SNLI training set (data/snli_train_100k.jsonl, SHA-256 0927a0b1…7f88; records without annotator consensus dropped). Only 9,488 pairs (28.4%) have a hard negative: the first contradiction hypothesis in file order for the exact same premise. The other 71.6% are trained with in-batch negatives only. Counts, the hard-negative rule and checksums are in the project's data/processed/MANIFEST.json. No MNLI data is used.

Training recipe

From runs/sup_seed42_bs128/config.json:

Hyperparameter	Value
Mode	supervised (entailment positives + contradiction hard negatives, in-batch negatives)
Base model	bert-base-uncased
Training pooling	[CLS], no MLP (pooling: cls, mlp_only_train: false)
Inference / eval pooling	cls
Use hard negatives	true
Batch size	128 (paper: 512)
Learning rate	5e-5, linear decay, warmup ratio 0.0
Optimizer	AdamW, weight decay 0.0, no gradient clipping (max_grad_norm: null)
Epochs	3 (780 steps)
Temperature τ	0.05
Max sequence length (training)	32
Dropout (hidden / attention)	0.1 / 0.1
Precision	fp16 (AMP)
Seed	42
Dev evaluation / checkpoint selection	every 50 steps on STS-B dev; best kept (step 350)
Hardware	1× Tesla T4 (Kaggle), ~7 min
Libraries	torch 2.10.0, transformers 5.18.0, Python 3.12

Evaluation

STS-B (sentence-transformers/stsb), protocol of the SimCSE paper (Appendix B): embed each sentence independently, cosine similarity, Spearman ρ against gold scores, no STS-B fine-tuning. Spearman ×100:

Model	Dev	Test	Alignment ↓	Uniformity ↓
This model	82.05	78.75	0.220	−3.362
Unsupervised companion (simcse-bert-base-snli-unsup)	76.92	68.33	0.305	−2.963
raw bert-base-uncased (mean pooling)	59.31	47.29	0.195	−1.650
SBERT-2019 (bert-base-nli-mean-tokens)	80.77	76.98	0.193	−3.049
SimCSE paper, sup BERT-base (reported)	86.20	84.25	—	—

• Dev is the best periodic dev score during training (used for checkpoint selection, so optimistically biased). Test was scored once on the kept checkpoint and reproduced after publishing by reloading this repo from the Hub into an empty cache (same value to floating-point precision).
• Alignment / uniformity (Wang & Isola, 2020) on STS-B dev, L2-normalized: alignment over the 264 dev pairs with gold ≥ 4; uniformity over the 2,910 unique dev sentences.
• Paraphrase retrieval over STS-B dev (264 gold ≥ 4 pairs, 2,910-sentence corpus): Recall@1 0.890, Recall@5 0.985, MRR 0.929.
• Ablation: the same recipe without hard negatives reaches 81.82 dev (+0.23 from hard negatives, vs. +1.3 in the paper; the difference is explained by the 28.4% hard-negative coverage).

Limitations

• Gap to the paper: −4.15 dev / −5.50 test. About three quarters of it is attributed to the data subset: 9.4× fewer training pairs from a single genre (SNLI only, no MNLI; 2.1 dev / ~3.1 test) and hard negatives for only 28.4% of anchors (1.0). The rest is untuned hyperparameters (batch 128 with the LR the paper tuned for 512; 0.3), dev-selection optimism on test (0.4) and single-seed noise (~0.7; the noise floor is about ±0.5). One configuration and one seed were run.
• Same topic, different event. Headlines that share a template but report different events are treated as near-paraphrases: "6.4-magnitude quake strikes off Indonesia" ↔ "6.9-magnitude quake strikes off Russia's Kuril Islands" (cosine 0.84, gold 1/5); "Military plane crashes in south France" → "…southeastern Turkey, 1 dead" (0.78, gold 1/5). SNLI contradictions are about scene content, so nothing teaches the model that changing a place, number or named entity changes the meaning.
• Domain. Trained only on English image-caption sentences of ≤ 32 tokens; news-style text is where most STS-B errors occur. Expect weaker results on long text, other languages, and specialized domains. Not evaluated on anything other than STS-B.
• Inherits the biases of BERT's pretraining data and of SNLI.

Citation

@inproceedings{gao2021simcse,
  title={SimCSE: Simple Contrastive Learning of Sentence Embeddings},
  author={Gao, Tianyu and Yao, Xingcheng and Chen, Danqi},
  booktitle={EMNLP},
  year={2021}
}