simcse-bert-base-snli-unsup

Unsupervised SimCSE (Gao, Yao & Chen, EMNLP 2021, Eq. 1) on top of bert-base-uncased, trained on sentences from a 100k-record SNLI subset as a course assignment (U2T02, Trends in Data Science). It maps a sentence to a 768-dimensional, L2-normalized vector; compare vectors with cosine similarity.

Team: Julio Cesar De Aquino Castellanos, Gustavo Alexander Fuentes Marin, Valeria Nicol Hernandez Leon, Ricardo Daniel Horta Sanchez, Jose Angel Pech Xool, Lorena Danae Perez Lopez.

This is a reproduction under a reduced budget, not the paper's released checkpoint: it is 8.5 Spearman points below the paper on STS-B test (see Limitations). The supervised companion model is pyrawn/simcse-bert-base-snli-sup (78.75 test) and is better on every metric below.

Usage

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("pyrawn/simcse-bert-base-snli-unsup")
emb = model.encode(["A man is playing a guitar.", "Someone plays an instrument."])
print(model.similarity(emb[0:1], emb[1:2]))  # cosine similarity

Architecture: Transformer (BERT-base, max 512 tokens) → Pooling(cls) (last-layer [CLS] hidden state, without the MLP used during training and without BERT's pretrained pooler) → Normalize.

Training data

Not the paper's data. The paper trains unsupervised SimCSE on 10⁶ sentences sampled from English Wikipedia; this model was trained on 165,529 unique sentences (premises and hypotheses, exact-string deduplication) extracted from a fixed 100k-record subset of the SNLI training set (data/snli_train_100k.jsonl, SHA-256 0927a0b1…7f88). Counts and checksums are in the project's data/processed/MANIFEST.json. SNLI sentences are short English image captions, a much narrower domain than Wikipedia. Labels are not used.

Training recipe

From runs/unsup_seed42_bs64/config.json:

Hyperparameter	Value
Mode	unsupervised (dropout noise as the only augmentation; in-batch negatives)
Base model	bert-base-uncased
Training pooling	[CLS] + MLP (pooling: cls, mlp_only_train: true)
Inference / eval pooling	cls_before_pooler ([CLS] without the MLP)
Batch size	64
Learning rate	3e-5, linear decay, warmup ratio 0.0
Optimizer	AdamW, weight decay 0.0, max grad norm 1.0
Epochs	1 (2,586 steps)
Temperature τ	0.05
Max sequence length (training)	32
Dropout (hidden / attention)	0.1 / 0.1
Same dropout mask for both views	false
Precision	fp16 (AMP)
Seed	42
Dev evaluation / checkpoint selection	every 125 steps on STS-B dev; best kept (step 125)
Hardware	1× Tesla T4 (Kaggle), ~10 min
Libraries	torch 2.10.0, transformers 5.18.0, Python 3.12

Evaluation

STS-B (sentence-transformers/stsb), protocol of the SimCSE paper (Appendix B): embed each sentence independently, cosine similarity, Spearman ρ against gold scores, no STS-B fine-tuning. Spearman ×100:

Model	Dev	Test	Alignment ↓	Uniformity ↓
This model	76.92	68.33	0.305	−2.963
Supervised companion (simcse-bert-base-snli-sup)	82.05	78.75	0.220	−3.362
raw bert-base-uncased (mean pooling)	59.31	47.29	0.195	−1.650
SBERT-2019 (bert-base-nli-mean-tokens)	80.77	76.98	0.193	−3.049
SimCSE paper, unsup BERT-base (reported)	82.50	76.85	—	—

• Dev is the best periodic dev score during training (used for checkpoint selection, so optimistically biased). Test was scored once on the kept checkpoint and reproduced after publishing by reloading this repo from the Hub into an empty cache (same value to floating-point precision).
• Alignment / uniformity (Wang & Isola, 2020) on STS-B dev, L2-normalized: alignment over the 264 dev pairs with gold ≥ 4; uniformity over the 2,910 unique dev sentences.
• Paraphrase retrieval over STS-B dev (264 gold ≥ 4 pairs, 2,910-sentence corpus): Recall@1 0.830, Recall@5 0.958, MRR 0.885.

Limitations

• Gap to the paper: −5.58 dev / −8.52 test. About four fifths of it is attributed to the training corpus (165k short SNLI captions instead of 10⁶ Wikipedia sentences). Dev peaked at the very first evaluation (step 125, 8,000 sentences seen) and then drifted down to 74.6, so more training on this corpus would not help. The test gap is larger than the dev gap partly because the selected checkpoint is an isolated dev maximum (1 point of dev-selection optimism). One configuration and one seed were run (no hyperparameter search); the single-seed noise floor is about ±1 point.
• Lexical-template failures. Without paraphrase supervision the model keeps sentences with the same syntactic frame and topic close even when the action/object differ: "The man is stirring the rice." retrieves "The man is buttering the bread." (cosine 0.80, gold 0.4/5) above "A person is mixing a pot of rice."; "cracking eggs into a bowl" ranks the true paraphrase "broke raw eggs into a bowl" only 6th.
• Same topic, different event. News headlines sharing a template ("6.4-magnitude quake strikes off Indonesia" vs. "6.9-magnitude quake strikes off Russia's Kuril Islands", cosine 0.84, gold 1/5) are treated as near-paraphrases. The model encodes topic and form more than numbers or named entities.
• Domain. Trained only on English image-caption sentences of ≤ 32 tokens. Expect weaker results on long text, other languages, and specialized domains. Not evaluated on anything other than STS-B.
• Inherits the biases of BERT's pretraining data and of SNLI.

Citation

@inproceedings{gao2021simcse,
  title={SimCSE: Simple Contrastive Learning of Sentence Embeddings},
  author={Gao, Tianyu and Yao, Xingcheng and Chen, Danqi},
  booktitle={EMNLP},
  year={2021}
}