asi commited on
Commit
894997f
1 Parent(s): 34e74e9

:books: add documentation

Browse files
Files changed (1) hide show
  1. README.md +2 -3
README.md CHANGED
@@ -50,9 +50,8 @@ card for more detailed information about the pre-training procedure.
50
 
51
  ## Fine-tuning
52
 
53
- The BERT model was pretrained on [BookCorpus](https://yknzhu.wixsite.com/mbweb), a dataset consisting of 11,038
54
- unpublished books and [English Wikipedia](https://en.wikipedia.org/wiki/English_Wikipedia) (excluding lists, tables and
55
- headers).
56
 
57
  ### Hyper parameters
58
 
 
50
 
51
  ## Fine-tuning
52
 
53
+ We fine-tune the model using a contrastive objective. Formally, we compute the cosine similarity from each possible sentence pairs from the batch.
54
+ We then apply the cross entropy loss by comparing with true pairs.
 
55
 
56
  ### Hyper parameters
57