karimepachecog/simcse-sup-snli100k

Sentence embedding model trained with SimCSE (Gao, Yao and Chen, 2021) from bert-base-uncased for U2T02.

Training data

A shared 100k-record subset of SNLI (snli_train_100k.jsonl), not the paper's Wikipedia or SNLI+MNLI training set.

• Mode: supervised
• Examples used: 33351 (1.0 of the constructed set)
• Unsupervised set: 165,529 unique SNLI sentences
• Supervised set: 33,351 entailment pairs; a contradiction hard negative exists for 9,488 of them (28.4%) and is masked out otherwise

Recipe

pooling at train	CLS + BERT pooler MLP
pooling at test	cls
batch size	128
learning rate	5e-05
temperature	0.05
epochs	3
max length	32
dropout	0.1
seed	42
hard negatives	True
hardware	cuda:Tesla T4 (14.6 GB)

Optimizer: AdamW, weight decay 0.0, warmup ratio 0.0, linear decay.

Metrics

STS-B Spearman x100, cosine similarity, no regressor.

split	Spearman
dev	78.03973124609854
test	74.73372610133005

Alignment (STS-B test, gold >= 4): 0.13581092655658722

Uniformity (STS-B test, t=2): -2.472869873046875

Limitations

• English only, and the encoder is uncased.
• Trained on SNLI captions, which are shorter and more concrete than the Wikipedia sentences used for unsupervised SimCSE in the paper.
• The supervised set is about an order of magnitude smaller than SNLI+MNLI, and most pairs have no contradiction hard negative.
• The supervised batch size is 128, not the paper's 512. The learning rate was retuned on the STS-B dev set at this batch: 1e-5 scored 74.28, 3e-5 scored 76.67, and 5e-5 scored 78.04. This checkpoint uses 5e-05.
• Not a substitute for the published SimCSE checkpoints if you need the paper's STS-B numbers.