Create README.md
Browse files
README.md
ADDED
@@ -0,0 +1,84 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
1 |
+
---
|
2 |
+
language: fr
|
3 |
+
license: mit
|
4 |
+
datasets:
|
5 |
+
- oscar
|
6 |
+
---
|
7 |
+
|
8 |
+
## Model description
|
9 |
+
|
10 |
+
CamemBERT is a state-of-the-art language model for French based on the RoBERTa model. It is now available on Hugging Face in 6 different versions with varying number of parameters, amount of pretraining data and pretraining data source domains.
|
11 |
+
|
12 |
+
## Evaluation
|
13 |
+
|
14 |
+
The model developers evaluated CamemBERT using four different downstream tasks for French: part-of-speech (POS) tagging, dependency parsing, named entity recognition (NER) and natural language inference (NLI).
|
15 |
+
|
16 |
+
## Limitations and bias
|
17 |
+
|
18 |
+
Significant research has explored bias and fairness issues with language models (see, e.g., [Sheng et al. (2021)](https://aclanthology.org/2021.acl-long.330.pdf) and [Bender et al. (2021)](https://dl.acm.org/doi/pdf/10.1145/3442188.3445922)).
|
19 |
+
|
20 |
+
This model was pretrinaed on a subcorpus of OSCAR multilingual corpus. Some of the limitations and risks associated with the OSCAR dataset, which are further detailed in the [OSCAR dataset card](https://huggingface.co/datasets/oscar), include the following:
|
21 |
+
|
22 |
+
> The quality of some OSCAR sub-corpora might be lower than expected, specifically for the lowest-resource languages.
|
23 |
+
|
24 |
+
> Constructed from Common Crawl, Personal and sensitive information might be present.
|
25 |
+
|
26 |
+
## Training data
|
27 |
+
|
28 |
+
OSCAR or Open Super-large Crawled Aggregated coRpus is a multilingual corpus obtained by language classification and filtering of the Common Crawl corpus using the Ungoliant architecture.
|
29 |
+
|
30 |
+
## How to use
|
31 |
+
|
32 |
+
-**Filling masks using pipeline**
|
33 |
+
|
34 |
+
```python
|
35 |
+
>>> from transformers import pipeline
|
36 |
+
>>> camembert_fill_mask = pipeline("fill-mask", model="camembert-base")
|
37 |
+
>>> results = camembert_fill_mask("Le camembert est <mask> :)")
|
38 |
+
|
39 |
+
>>> result
|
40 |
+
[{'score': 0.49091097712516785,
|
41 |
+
'token': 7200,
|
42 |
+
'token_str': 'délicieux',
|
43 |
+
'sequence': 'Le camembert est délicieux :)'},
|
44 |
+
{'score': 0.1055697426199913,
|
45 |
+
'token': 2183,
|
46 |
+
'token_str': 'excellent',
|
47 |
+
'sequence': 'Le camembert est excellent :)'},
|
48 |
+
{'score': 0.03453319892287254,
|
49 |
+
'token': 26202,
|
50 |
+
'token_str': 'succulent',
|
51 |
+
'sequence': 'Le camembert est succulent :)'},
|
52 |
+
{'score': 0.03303128108382225,
|
53 |
+
'token': 528,
|
54 |
+
'token_str': 'meilleur',
|
55 |
+
'sequence': 'Le camembert est meilleur :)'},
|
56 |
+
{'score': 0.030076386407017708,
|
57 |
+
'token': 1654,
|
58 |
+
'token_str': 'parfait',
|
59 |
+
'sequence': 'Le camembert est parfait :)'}]
|
60 |
+
```
|
61 |
+
|
62 |
+
-**Extract contextual embedding features from Camembert output**
|
63 |
+
|
64 |
+
```python
|
65 |
+
import torch
|
66 |
+
|
67 |
+
>>> tokenized_sentence = tokenizer.tokenize("J'aime le camembert !")
|
68 |
+
>>> encoded_sentence = tokenizer.encode(tokenized_sentence)
|
69 |
+
# Can be done in one step : tokenize.encode("J'aime le camembert !")
|
70 |
+
|
71 |
+
>>> tokenized_sentence
|
72 |
+
['▁J', "'", 'aime', '▁le', '▁ca', 'member', 't', '▁!']
|
73 |
+
>>> encoded_sentence
|
74 |
+
[5, 121, 11, 660, 16, 730, 25543, 110, 83, 6]
|
75 |
+
|
76 |
+
encoded_sentence = torch.tensor(encoded_sentence).unsqueeze(0)
|
77 |
+
embeddings, _ = camembert(encoded_sentence)
|
78 |
+
# embeddings.detach()
|
79 |
+
# embeddings.size torch.Size([1, 10, 768])
|
80 |
+
# tensor([[[-0.0254, 0.0235, 0.1027, ..., -0.1459, -0.0205, -0.0116],
|
81 |
+
# [ 0.0606, -0.1811, -0.0418, ..., -0.1815, 0.0880, -0.0766],
|
82 |
+
# [-0.1561, -0.1127, 0.2687, ..., -0.0648, 0.0249, 0.0446],
|
83 |
+
# ...,
|
84 |
+
```
|