maj readme

357942a 3 months ago

No virus

8.25 kB

	---
	language: fr
	license: mit
	library_name: sentence-transformers
	pipeline_tag: feature-extraction
	tags:
	- sentence-transformers
	- feature-extraction
	- sentence-similarity
	- transformers
	datasets:
	- stsb_multi_mt
	metrics:
	- pearsonr
	base_model: cmarkea/distilcamembert-base
	model-index:
	- name: sts-distilcamembert-base
	results:
	- task:
	name: Sentence Similarity
	type: sentence-similarity
	dataset:
	name: STSb French
	type: stsb_multi_mt
	args: fr
	metrics:
	- name: Pearson Correlation - stsb_multi_mt fr
	type: pearsonr
	value: 0.8165
	---

	## Description

	Ce modèle [sentence-transformers](https://www.SBERT.net) a été obtenu en finetunant le modèle
	[`cmarkea/distilcamembert-base`](https://huggingface.co/cmarkea/distilcamembert-base) à l'aide de la librairie
	[sentence-transformers](https://www.SBERT.net).

	Il permet d'encoder une phrase ou un pararaphe (514 tokens maximum) en un vecteur de dimension 768.

	Le modèle [DistilCamemBERT](https://huggingface.co/papers/2205.11111) sur lequel il est basé est une distillation du
	modèlel [CamemBERT](https://arxiv.org/abs/1911.03894) permettant de diviser par deux le nombre de paramètres du modèle
	et améliorer le temps d'inférence.

	## Utilisation via la librairie `sentence-transformers`

	```
	pip install -U sentence-transformers
	```

	```python
	from sentence_transformers import SentenceTransformer
	sentences = ["Ceci est un exemple", "deuxième exemple"]

	model = SentenceTransformer('h4c5/sts-distilcamembert-base')
	embeddings = model.encode(sentences)
	print(embeddings)
	```


	## Utilisation via la librairie `transformers`

	```
	pip install -U transformers
	```

	```python
	from transformers import AutoTokenizer, AutoModel
	import torch

	tokenizer = AutoTokenizer.from_pretrained("h4c5/sts-distilcamembert-base")
	model = AutoModel.from_pretrained("h4c5/sts-distilcamembert-base")
	model.eval()


	# Mean Pooling
	def mean_pooling(model_output, attention_mask):
	token_embeddings = model_output[
	0
	] # First element of model_output contains all token embeddings
	input_mask_expanded = (
	attention_mask.unsqueeze(-1).expand(token_embeddings.size()).float()
	)
	return torch.sum(token_embeddings * input_mask_expanded, 1) / torch.clamp(
	input_mask_expanded.sum(1), min=1e-9
	)

	# Tokenization et calcul des embeddings des tokens
	encoded_input = tokenizer(sentences, padding=True, truncation=True, return_tensors="pt")
	model_output = model(**encoded_input)

	# Mean pooling
	sentence_embeddings = mean_pooling(model_output, encoded_input["attention_mask"])

	print(sentence_embeddings)
	```


	## Evaluation

	Le modèle a été évalué sur le jeu de données [STSb fr](https://huggingface.co/datasets/stsb_multi_mt) :

	```python
	from datasets import load_dataset
	from sentence_transformers import InputExample, evaluation


	def dataset_to_input_examples(dataset):
	return [
	InputExample(
	texts=[example["sentence1"], example["sentence2"]],
	label=example["similarity_score"] / 5.0,
	)
	for example in dataset
	]


	sts_test_dataset = load_dataset("stsb_multi_mt", name="fr", split="test")
	sts_test_examples = dataset_to_input_examples(sts_test_dataset)

	sts_test_evaluator = evaluation.EmbeddingSimilarityEvaluator.from_input_examples(
	sts_test_examples, name="sts-test"
	)

	sts_test_evaluator(model, ".")
	```

	### Résultats

	Ci-dessous, les résultats de l'évaluation du modèle sur le jeu données [`stsb_multi_mt`](https://huggingface.co/datasets/stsb_multi_mt)
	(données `fr`, split `test`)

	\| Model \| Pearson Correlation \| Paramètres \|
	\| :--------------------------------------------------------------------------------------------------------------------------------------------- \| :-----------------: \| ---------: \|
	\| [`h4c5/sts-camembert-base`](https://huggingface.co/h4c5/sts-camembert-base) \| 0.837 \| 110M \|
	\| [`Lajavaness/sentence-camembert-base`](https://huggingface.co/Lajavaness/sentence-camembert-base) \| 0.835 \| 110M \|
	\| [`inokufu/flaubert-base-uncased-xnli-sts`](https://huggingface.co/inokufu/flaubert-base-uncased-xnli-sts) \| 0.828 \| 137M \|
	\| [`h4c5/sts-distilcamembert-base`](https://huggingface.co/h4c5/sts-distilcamembert-base) \| 0.817 \| 68M \|
	\| [`sentence-transformers/distiluse-base-multilingual-cased-v2`](https://huggingface.co/sentence-transformers/distiluse-base-multilingual-cased) \| 0.786 \| 135M \|



	## Training
	The model was trained with the parameters:

	DataLoader:

	`torch.utils.data.dataloader.DataLoader` of length 180 with parameters:
	```
	{'batch_size': 32, 'sampler': 'torch.utils.data.sampler.RandomSampler', 'batch_sampler': 'torch.utils.data.sampler.BatchSampler'}
	```

	Loss:

	`sentence_transformers.losses.CosineSimilarityLoss.CosineSimilarityLoss`

	Parameters of the `fit()` method:
	```
	{
	"epochs": 10,
	"evaluation_steps": 1000,
	"evaluator": "sentence_transformers.evaluation.EmbeddingSimilarityEvaluator.EmbeddingSimilarityEvaluator",
	"max_grad_norm": 1,
	"optimizer_class": "<class 'torch.optim.adamw.AdamW'>",
	"optimizer_params": {
	"lr": 2e-05
	},
	"scheduler": "WarmupLinear",
	"steps_per_epoch": null,
	"warmup_steps": 500,
	"weight_decay": 0.01
	}
	```


	## Full Model Architecture

	```
	SentenceTransformer(
	(0): Transformer({'max_seq_length': 512, 'do_lower_case': False}) with Transformer model: CamembertModel
	(1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
	)
	```

	## Citing

	@inproceedings{reimers-2019-sentence-bert,
	title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
	author = "Reimers, Nils and Gurevych, Iryna",
	booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
	month = "11",
	year = "2019",
	publisher = "Association for Computational Linguistics",
	journal={"https://arxiv.org/abs/1908.10084"},
	}

	@inproceedings{sanh2019distilbert,
	title={DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter},
	author={Sanh, Victor and Debut, Lysandre and Chaumond, Julien and Wolf, Thomas},
	booktitle={NeurIPS EMC^2 Workshop},
	journal={https://arxiv.org/abs/1910.01108},
	year={2019}
	}

	@inproceedings{martin2020camembert,
	title={CamemBERT: a Tasty French Language Model},
	author={Martin, Louis and Muller, Benjamin and Su{\'a}rez, Pedro Javier Ortiz and Dupont, Yoann and Romary, Laurent and de la Clergerie, {\'E}ric Villemonte and Seddah, Djam{\'e} and Sagot, Beno{\^\i}t},
	booktitle={Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics},
	journal={https://arxiv.org/abs/1911.03894},
	year={2020}
	}

	@inproceedings{delestre:hal-03674695,
	TITLE = {{DistilCamemBERT : une distillation du mod{\`e}le fran{\c c}ais CamemBERT}},
	AUTHOR = {Delestre, Cyrile and Amar, Abibatou},
	URL = {https://hal.archives-ouvertes.fr/hal-03674695},
	BOOKTITLE = {{CAp (Conf{\'e}rence sur l'Apprentissage automatique)}},
	ADDRESS = {Vannes, France},
	YEAR = {2022},
	MONTH = Jul,
	KEYWORDS = {NLP ; Transformers ; CamemBERT ; Distillation},
	PDF = {https://hal.archives-ouvertes.fr/hal-03674695/file/cap2022.pdf},
	HAL_ID = {hal-03674695},
	HAL_VERSION = {v1},
	journal={https://arxiv.org/abs/2205.11111},
	}