Hugging Face's logo

FacebookAI
/
xlm-mlm-enfr-1024

Fill-Mask Transformers PyTorch TensorFlow multilingual English French xlm Inference Endpoints
Model card Files Community
3
sgugger Marissa commited on
Commit
31a5a94
•
1 Parent(s): 5f1f914

Add model card (#1)

Browse files

- Add model card (fd7f5764d1c0ae82730c086daca19387959f1b62)
- Update README.md (4b9459f2ebd9d4d4636ffa193fdece0d959e2ae3)
- Update README.md (480cc05f4e4ebb9ac7787a3fa60d65aae2293ea5)
- Update README.md (aa9af656f9dc8d6197c82b5eefbe445a85a2c3ea)
- Add license info (4ac51db28266e4fb627b6d94ae2ad4f41338ccf0)


Co-authored-by: Marissa Gerchick <Marissa@users.noreply.huggingface.co>

Files changed (1) hide show
  1. README.md +127 -0
README.md ADDED
@@ -0,0 +1,127 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ language:
3
+ - multilingual
4
+ - en
5
+ - fr
6
+ license: cc-by-nc-4.0
7
+ ---
8
+
9
+ # xlm-mlm-enfr-1024
10
+
11
+ # Table of Contents
12
+
13
+ 1. [Model Details](#model-details)
14
+ 2. [Uses](#uses)
15
+ 3. [Bias, Risks, and Limitations](#bias-risks-and-limitations)
16
+ 4. [Training](#training)
17
+ 5. [Evaluation](#evaluation)
18
+ 6. [Environmental Impact](#environmental-impact)
19
+ 7. [Technical Specifications](#technical-specifications)
20
+ 8. [Citation](#citation)
21
+ 9. [Model Card Authors](#model-card-authors)
22
+ 10. [How To Get Started With the Model](#how-to-get-started-with-the-model)
23
+
24
+
25
+ # Model Details
26
+
27
+ The XLM model was proposed in [Cross-lingual Language Model Pretraining](https://arxiv.org/abs/1901.07291) by Guillaume Lample, Alexis Conneau. xlm-mlm-enfr-1024 is a transformer pretrained using a masked language modeling (MLM) objective for English-French. This model uses language embeddings to specify the language used at inference. See the [Hugging Face Multilingual Models for Inference docs](https://huggingface.co/docs/transformers/v4.20.1/en/multilingual#xlm-with-language-embeddings) for further details.
28
+
29
+ ## Model Description
30
+
31
+ - **Developed by:** Guillaume Lample, Alexis Conneau, see [associated paper](https://arxiv.org/abs/1901.07291)
32
+ - **Model type:** Language model
33
+ - **Language(s) (NLP):** English-French
34
+ - **License:** CC-BY-NC-4.0
35
+ - **Related Models:** [xlm-clm-ende-1024](https://huggingface.co/xlm-clm-enfr-1024), [xlm-clm-ende-1024](https://huggingface.co/xlm-clm-ende-1024), [xlm-mlm-ende-1024](https://huggingface.co/xlm-mlm-ende-1024), [xlm-mlm-enro-1024](https://huggingface.co/xlm-mlm-enro-1024)
36
+ - **Resources for more information:**
37
+ - [Associated paper](https://arxiv.org/abs/1901.07291)
38
+ - [GitHub Repo](https://github.com/facebookresearch/XLM)
39
+ - [Hugging Face Multilingual Models for Inference docs](https://huggingface.co/docs/transformers/v4.20.1/en/multilingual#xlm-with-language-embeddings)
40
+
41
+ # Uses
42
+
43
+ ## Direct Use
44
+
45
+ The model is a language model. The model can be used for masked language modeling.
46
+
47
+ ## Downstream Use
48
+
49
+ To learn more about this task and potential downstream uses, see the Hugging Face [fill mask docs](https://huggingface.co/tasks/fill-mask) and the [Hugging Face Multilingual Models for Inference](https://huggingface.co/docs/transformers/v4.20.1/en/multilingual#xlm-with-language-embeddings) docs.
50
+
51
+ ## Out-of-Scope Use
52
+
53
+ The model should not be used to intentionally create hostile or alienating environments for people.
54
+
55
+ # Bias, Risks, and Limitations
56
+
57
+ Significant research has explored bias and fairness issues with language models (see, e.g., [Sheng et al. (2021)](https://aclanthology.org/2021.acl-long.330.pdf) and [Bender et al. (2021)](https://dl.acm.org/doi/pdf/10.1145/3442188.3445922)).
58
+
59
+ ## Recommendations
60
+
61
+ Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model.
62
+
63
+ # Training
64
+
65
+ The model developers write:
66
+
67
+ > In all experiments, we use a Transformer architecture with 1024 hidden units, 8 heads, GELU activations (Hendrycks and Gimpel, 2016), a dropout rate of 0.1 and learned positional embeddings. We train our models with the Adam op- timizer (Kingma and Ba, 2014), a linear warm- up (Vaswani et al., 2017) and learning rates varying from 10^−4 to 5.10^−4.
68
+
69
+ See the [associated paper](https://arxiv.org/pdf/1901.07291.pdf) for links, citations, and further details on the training data and training procedure.
70
+
71
+ The model developers also write that:
72
+
73
+ > If you use these models, you should use the same data preprocessing / BPE codes to preprocess your data.
74
+
75
+ See the associated [GitHub Repo](https://github.com/facebookresearch/XLM#ii-cross-lingual-language-model-pretraining-xlm) for further details.
76
+
77
+ # Evaluation
78
+
79
+ ## Testing Data, Factors & Metrics
80
+
81
+ The model developers evaluated the model on the [WMT'14 English-French](https://huggingface.co/datasets/wmt14) dataset using the [BLEU metric](https://huggingface.co/spaces/evaluate-metric/bleu). See the [associated paper](https://arxiv.org/pdf/1901.07291.pdf) for further details on the testing data, factors and metrics.
82
+
83
+ ## Results
84
+
85
+ For xlm-mlm-enfr-1024 results, see Table 1 and Table 2 of the [associated paper](https://arxiv.org/pdf/1901.07291.pdf).
86
+
87
+ # Environmental Impact
88
+
89
+ Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
90
+
91
+ - **Hardware Type:** More information needed
92
+ - **Hours used:** More information needed
93
+ - **Cloud Provider:** More information needed
94
+ - **Compute Region:** More information needed
95
+ - **Carbon Emitted:** More information needed
96
+
97
+ # Technical Specifications
98
+
99
+ The model developers write:
100
+
101
+ > We implement all our models in PyTorch (Paszke et al., 2017), and train them on 64 Volta GPUs for the language modeling tasks, and 8 GPUs for the MT tasks. We use float16 operations to speed up training and to reduce the memory usage of our models.
102
+
103
+ See the [associated paper](https://arxiv.org/pdf/1901.07291.pdf) for further details.
104
+
105
+ # Citation
106
+
107
+ **BibTeX:**
108
+
109
+ ```bibtex
110
+ @article{lample2019cross,
111
+ title={Cross-lingual language model pretraining},
112
+ author={Lample, Guillaume and Conneau, Alexis},
113
+ journal={arXiv preprint arXiv:1901.07291},
114
+ year={2019}
115
+ }
116
+ ```
117
+
118
+ **APA:**
119
+ - Lample, G., & Conneau, A. (2019). Cross-lingual language model pretraining. arXiv preprint arXiv:1901.07291.
120
+
121
+ # Model Card Authors
122
+
123
+ This model card was written by the team at Hugging Face.
124
+
125
+ # How to Get Started with the Model
126
+
127
+ More information needed. This model uses language embeddings to specify the language used at inference. See the [Hugging Face Multilingual Models for Inference docs](https://huggingface.co/docs/transformers/v4.20.1/en/multilingual#xlm-with-language-embeddings) for further details.