nrnmnrn commited on
Commit
ce4dc4f
1 Parent(s): 94bc3ff

Create README.md

Browse files
Files changed (1) hide show
  1. README.md +84 -0
README.md ADDED
@@ -0,0 +1,84 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ language: fr
3
+ license: mit
4
+ datasets:
5
+ - oscar
6
+ ---
7
+
8
+ ## Model description
9
+
10
+ CamemBERT is a state-of-the-art language model for French based on the RoBERTa model. It is now available on Hugging Face in 6 different versions with varying number of parameters, amount of pretraining data and pretraining data source domains.
11
+
12
+ ## Evaluation
13
+
14
+ The model developers evaluated CamemBERT using four different downstream tasks for French: part-of-speech (POS) tagging, dependency parsing, named entity recognition (NER) and natural language inference (NLI).
15
+
16
+ ## Limitations and bias
17
+
18
+ Significant research has explored bias and fairness issues with language models (see, e.g., [Sheng et al. (2021)](https://aclanthology.org/2021.acl-long.330.pdf) and [Bender et al. (2021)](https://dl.acm.org/doi/pdf/10.1145/3442188.3445922)).
19
+
20
+ This model was pretrinaed on a subcorpus of OSCAR multilingual corpus. Some of the limitations and risks associated with the OSCAR dataset, which are further detailed in the [OSCAR dataset card](https://huggingface.co/datasets/oscar), include the following:
21
+
22
+ > The quality of some OSCAR sub-corpora might be lower than expected, specifically for the lowest-resource languages.
23
+
24
+ > Constructed from Common Crawl, Personal and sensitive information might be present.
25
+
26
+ ## Training data
27
+
28
+ OSCAR or Open Super-large Crawled Aggregated coRpus is a multilingual corpus obtained by language classification and filtering of the Common Crawl corpus using the Ungoliant architecture.
29
+
30
+ ## How to use
31
+
32
+ -**Filling masks using pipeline**
33
+
34
+ ```python
35
+ >>> from transformers import pipeline
36
+ >>> camembert_fill_mask = pipeline("fill-mask", model="camembert-base")
37
+ >>> results = camembert_fill_mask("Le camembert est <mask> :)")
38
+
39
+ >>> result
40
+ [{'score': 0.49091097712516785,
41
+ 'token': 7200,
42
+ 'token_str': 'délicieux',
43
+ 'sequence': 'Le camembert est délicieux :)'},
44
+ {'score': 0.1055697426199913,
45
+ 'token': 2183,
46
+ 'token_str': 'excellent',
47
+ 'sequence': 'Le camembert est excellent :)'},
48
+ {'score': 0.03453319892287254,
49
+ 'token': 26202,
50
+ 'token_str': 'succulent',
51
+ 'sequence': 'Le camembert est succulent :)'},
52
+ {'score': 0.03303128108382225,
53
+ 'token': 528,
54
+ 'token_str': 'meilleur',
55
+ 'sequence': 'Le camembert est meilleur :)'},
56
+ {'score': 0.030076386407017708,
57
+ 'token': 1654,
58
+ 'token_str': 'parfait',
59
+ 'sequence': 'Le camembert est parfait :)'}]
60
+ ```
61
+
62
+ -**Extract contextual embedding features from Camembert output**
63
+
64
+ ```python
65
+ import torch
66
+
67
+ >>> tokenized_sentence = tokenizer.tokenize("J'aime le camembert !")
68
+ >>> encoded_sentence = tokenizer.encode(tokenized_sentence)
69
+ # Can be done in one step : tokenize.encode("J'aime le camembert !")
70
+
71
+ >>> tokenized_sentence
72
+ ['▁J', "'", 'aime', '▁le', '▁ca', 'member', 't', '▁!']
73
+ >>> encoded_sentence
74
+ [5, 121, 11, 660, 16, 730, 25543, 110, 83, 6]
75
+
76
+ encoded_sentence = torch.tensor(encoded_sentence).unsqueeze(0)
77
+ embeddings, _ = camembert(encoded_sentence)
78
+ # embeddings.detach()
79
+ # embeddings.size torch.Size([1, 10, 768])
80
+ # tensor([[[-0.0254, 0.0235, 0.1027, ..., -0.1459, -0.0205, -0.0116],
81
+ # [ 0.0606, -0.1811, -0.0418, ..., -0.1815, 0.0880, -0.0766],
82
+ # [-0.1561, -0.1127, 0.2687, ..., -0.0648, 0.0249, 0.0446],
83
+ # ...,
84
+ ```