Create README.md

README.md

@@ -0,0 +1,178 @@
+---
+language: 
+- multilingual
+- af
+- am
+- ar
+- as
+- az
+- be
+- bg
+- bn
+- br
+- bs
+- ca
+- cs
+- cy
+- da
+- de
+- el
+- en
+- eo
+- es
+- et
+- eu
+- fa
+- fi
+- fr
+- fy
+- ga
+- gd
+- gl
+- gu
+- ha
+- he
+- hi
+- hr
+- hu
+- hy
+- id
+- is
+- it
+- ja
+- jv
+- ka
+- kk
+- km
+- kn
+- ko
+- ku
+- ky
+- la
+- lo
+- lt
+- lv
+- mg
+- mk
+- ml
+- mn
+- mr
+- ms
+- my
+- ne
+- nl
+- no
+- om
+- or
+- pa
+- pl
+- ps
+- pt
+- ro
+- ru
+- sa
+- sd
+- si
+- sk
+- sl
+- so
+- sq
+- sr
+- su
+- sv
+- sw
+- ta
+- te
+- th
+- tl
+- tr
+- ug
+- uk
+- ur
+- uz
+- vi
+- xh
+- yi
+- zh
+license: mit
+---
+
+# XLM-V (Base-sized model)
+
+XLM-V is multilingual language model with a one million token vocabulary trained on 2.5TB of data from Common Crawl (same as XLM-R).
+It was introduced in the [XLM-V: Overcoming the Vocabulary Bottleneck in Multilingual Masked Language Models](https://arxiv.org/abs/2301.10472)
+paper by Davis Liang, Hila Gonen, Yuning Mao, Rui Hou, Naman Goyal, Marjan Ghazvininejad, Luke Zettlemoyer and Madian Khabsa.
+
+**Disclaimer**: The team releasing XLM-V did not write a model card for this model so this model card has been written by the Hugging Face team. [This repository](https://github.com/stefan-it/xlm-v-experiments) documents all necessary integeration steps.
+
+## Model description
+
+From the abstract of the XLM-V paper:
+
+> Large multilingual language models typically rely on a single vocabulary shared across 100+ languages.
+> As these models have increased in parameter count and depth, vocabulary size has remained largely unchanged.
+> This vocabulary bottleneck limits the representational capabilities of multilingual models like XLM-R.
+> In this paper, we introduce a new approach for scaling to very large multilingual vocabularies by
+> de-emphasizing token sharing between languages with little lexical overlap and assigning vocabulary capacity
+> to achieve sufficient coverage for each individual language. Tokenizations using our vocabulary are typically
+> more semantically meaningful and shorter compared to XLM-R. Leveraging this improved vocabulary, we train XLM-V,
+> a multilingual language model with a one million token vocabulary. XLM-V outperforms XLM-R on every task we
+> tested on ranging from natural language inference (XNLI), question answering (MLQA, XQuAD, TyDiQA), and
+> named entity recognition (WikiAnn) to low-resource tasks (Americas NLI, MasakhaNER).
+
+## Usage
+
+You can use this model directly with a pipeline for masked language modeling:
+
+```python
+>>> from transformers import pipeline
+>>> unmasker = pipeline('fill-mask', model='facebook/xlm-v-base')
+>>> unmasker("Paris is the <mask> of France.")
+
+[{'score': 0.9286897778511047,
+  'token': 133852,
+  'token_str': 'capital',
+  'sequence': 'Paris is the capital of France.'},
+ {'score': 0.018073994666337967,
+  'token': 46562,
+  'token_str': 'Capital',
+  'sequence': 'Paris is the Capital of France.'},
+ {'score': 0.013238662853837013,
+  'token': 8696,
+  'token_str': 'centre',
+  'sequence': 'Paris is the centre of France.'},
+ {'score': 0.010450296103954315,
+  'token': 550136,
+  'token_str': 'heart',
+  'sequence': 'Paris is the heart of France.'},
+ {'score': 0.005028395913541317,
+  'token': 60041,
+  'token_str': 'center',
+  'sequence': 'Paris is the center of France.'}]
+```
+
+## Bias, Risks, and Limitations
+
+Please refer to the model card of [XLM-R](https://huggingface.co/xlm-roberta-base), because XLM-V has a similar architecture
+and has been trained on similar training data.
+
+### BibTeX entry and citation info
+
+```bibtex
+@ARTICLE{2023arXiv230110472L,
+       author = {{Liang}, Davis and {Gonen}, Hila and {Mao}, Yuning and {Hou}, Rui and {Goyal}, Naman and {Ghazvininejad}, Marjan and {Zettlemoyer}, Luke and {Khabsa}, Madian},
+        title = "{XLM-V: Overcoming the Vocabulary Bottleneck in Multilingual Masked Language Models}",
+      journal = {arXiv e-prints},
+     keywords = {Computer Science - Computation and Language, Computer Science - Machine Learning},
+         year = 2023,
+        month = jan,
+          eid = {arXiv:2301.10472},
+        pages = {arXiv:2301.10472},
+          doi = {10.48550/arXiv.2301.10472},
+archivePrefix = {arXiv},
+       eprint = {2301.10472},
+ primaryClass = {cs.CL},
+       adsurl = {https://ui.adsabs.harvard.edu/abs/2023arXiv230110472L},
+      adsnote = {Provided by the SAO/NASA Astrophysics Data System}
+}
+```