Train a Sentence-CamemBERT
Tags: python, datascience, embedding, nlp
Published: 2024-03-09 11:59:30

The [CamemBERT](https://camembert-model.fr) model is a state-of-the-art language model for modeling the
French language.

It is a [RoBERTa](https://arxiv.org/abs/1907.11692) model that has been trained on a large number of French texts and can be easily adapted to a large number of tasks thanks to finetuning.

Here we're going to to finetune the model for sentence embedding.

## Sentence-BERT

The output of a BERT model is an embedding vector for each token. To obtain an embedding of the text as a whole, we need to define a transformation strategy to go from individual token embeddings to an embedding vector for the sentence as a whole.

The simplest and most effective strategy is simply to take the average of the token embeddings. 
This strategy is known as _mean pooling_.

![Sentence-BERT diagram : token embeddings are average in a mean pooling layer to get the sentence embedding](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/oc8roxd60aa2iormvea6.png)

If you'd like to find out more about the strategies that have been considered, take a look at this paper: [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://arxiv.org/pdf/1908.10084.pdf).

## Finetuning a BERT model into a Sentence-BERT model

The authors of the paper mentioned above have built a Python library called [sentence-transformers](https://www.sbert.net/), for manipulating Sentence-BERT models.

We'll use it to obtain a Sentence-CamemBERT model from a CamemBERT model available on [huggingface](https://huggingface.co/almanach/camembert-base).

### Prerequisites

We will be using the following packages:

```
datasets
sentence-transformers
```

### Training data

First of all, we're going to retrieve the training data.
We will use the French part of the dataset [STSb Multi MT](https://huggingface.co/datasets/stsb_multi_mt).
This is a dataset containing pairs of sentences and a score between 0 and 5 representing the similarity between the two sentences.

```python
from datasets import load_dataset

sts_train_dataset = load_dataset("stsb_multi_mt", name="fr", split="train")
sts_dev_dataset = load_dataset("stsb_multi_mt", name="fr", split="dev")
sts_test_dataset = load_dataset("stsb_multi_mt", name="fr", split="test")
```

We'll then convert the retrieved data into InputExample objects that can be used for training.

```python
from typing import List
from sentence_transformers import InputExample

def dataset_to_input_examples(dataset) -> List[InputExample]:
    return [
    InputExample(
        texts=[example["sentence1"], example["sentence2"]],
        label=example["similarity_score"] / 5.0,
    )
    for example in dataset
]

sts_train_examples = dataset_to_input_examples(sts_train_dataset)
sts_dev_examples = dataset_to_input_examples(sts_dev_dataset)
sts_test_examples = dataset_to_input_examples(sts_test_dataset)
```

We will use the CamemBERT model named [`almanach/camembert-base`](https://huggingface.co/almanach/camembert-base) for finetuning:

```python
from sentence_transformers import evaluation, losses, SentenceTransformer
from torch.utils.data import DataLoader

batch_size = 32

model = SentenceTransformer("almanach/camembert-base")

train_dataloader = DataLoader(sts_train_examples, shuffle=True, batch_size=batch_size)
train_loss = losses.CosineSimilarityLoss(model=model)
```

We use cosine-similarity loss objective to train the model.

Finally, an evaluator is built to monitor the model's performance on the dev dataset during training.

```python
sts_dev_evaluator = evaluation.EmbeddingSimilarityEvaluator.from_input_examples(
    sts_dev_examples, name="sts-dev"
)
```

We can now start training the model:

```python
model.fit(
    train_objectives=[(train_dataloader, train_loss)],
    evaluator=sts_dev_evaluator,
    epochs=10,
    warmup_steps=500,
    save_best_model=True,
)
```

### Model evaluation

Once training is complete, you can measure the model's performance on the test data set you've kept
away:

```python
sts_test_evaluator = evaluation.EmbeddingSimilarityEvaluator.from_input_examples(
    sts_test_examples, name="sts-test"
)

sts_test_evaluator(model, ".")
```

I get a Pearson correlation of `0.837` which is at the same level as the Sentence-CamemBERT I found on huggingface :

| Model | Pearson Correlation | Parameters |
| :--------------------------------------------------------------------------------------------------------------------------------------------- | :-----------------: | ---------: |
| [`h4c5/sts-camembert-base`](https://huggingface.co/h4c5/sts-camembert-base) | **0.837** | 110M |
| [`Lajavaness/sentence-camembert-base`](https://huggingface.co/Lajavaness/sentence-camembert-base) | 0.835 | 110M |
| [`inokufu/flaubert-base-uncased-xnli-sts`](https://huggingface.co/inokufu/flaubert-base-uncased-xnli-sts) | 0.828 | 137M |
| [`h4c5/sts-distilcamembert-base`](https://huggingface.co/h4c5/sts-distilcamembert-base) | 0.817 | 68M |
| [`sentence-transformers/distiluse-base-multilingual-cased-v2`](https://huggingface.co/sentence-transformers/distiluse-base-multilingual-cased) | 0.786 | 135M |

## Sentence-BERT model distilled

As you may have noticed in the table above, I've also trained a Sentence-CamemBERT model that's about half the size (68M parameters vs. 110M) and yet performs very well: [`h4c5/sts-distilcamembert-base`](https://huggingface.co/h4c5/sts-distilcamembert-base).

This is in fact a model obtained by following the above procedure but starting from the distilled CamemBERT model: [cmarkea/distilcamembert-base](https://huggingface.co/cmarkea/distilcamembert-base).

This so-called "distilled" model was obtained by removing half of the layer of the CamemBERT base model and training it to maintain its performance.

To find out more about the distillation process, please consult the following papers:

* [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://arxiv.org/abs/1910.01108)
* [DistilCamemBERT: a distilled version of the French CamemBERT](https://hal.archives-ouvertes.fr/hal-03674695)

Et voilà. You can find my two Sentence-CamemBERT models on huggingface :

* [h4c5/sts-camembert-base](https://huggingface.co/h4c5/sts-camembert-base)
* [h4c5/sts-distilcamembert-base](https://huggingface.co/h4c5/sts-distilcamembert-base)