Update README.md

README.md

@@ -1,12 +1,155 @@
 ---
-language: fi
-license: CC-BY 4.0
+language:
+- fi
+license: apache-2.0
 tags:
 - finnish
 - roberta
-pipeline_tag: fill-mask
+datasets:
+- mc4
 widget:
-- text: "Hän lähti ajamaan <mask> töihin."
+- text: "Moikka olen <mask> kielimalli."
+
 ---
 
-# Finnish roberta-large
+# NOTE: We have trained newer and better Finnish RoBERTa large model which can be found from different repository: [https://huggingface.co/Finnish-NLP/roberta-large-finnish](https://huggingface.co/Finnish-NLP/roberta-large-finnish). Our future Finnish models will be available at the [Finnish-NLP](https://huggingface.co/Finnish-NLP) Hugging Face organization
+
+
+# RoBERTa large model for Finnish
+
+Pretrained model on Finnish language using a masked language modeling (MLM) objective. It was introduced in
+[this paper](https://arxiv.org/abs/1907.11692) and first released in
+[this repository](https://github.com/pytorch/fairseq/tree/master/examples/roberta). This model is case-sensitive: it
+makes a difference between finnish and Finnish.
+
+## Model description
+
+RoBERTa is a transformers model pretrained on a large corpus of Finnish data in a self-supervised fashion. This means
+it was pretrained on the raw texts only, with no humans labelling them in any way (which is why it can use lots of
+publicly available data) with an automatic process to generate inputs and labels from those texts. 
+
+More precisely, it was pretrained with the Masked language modeling (MLM) objective. Taking a sentence, the model
+randomly masks 15% of the words in the input then run the entire masked sentence through the model and has to predict
+the masked words. This is different from traditional recurrent neural networks (RNNs) that usually see the words one
+after the other, or from autoregressive models like GPT which internally mask the future tokens. It allows the model to
+learn a bidirectional representation of the sentence.
+
+This way, the model learns an inner representation of the Finnish language that can then be used to extract features
+useful for downstream tasks: if you have a dataset of labeled sentences for instance, you can train a standard
+classifier using the features produced by the RoBERTa model as inputs.
+
+## Intended uses & limitations
+
+You can use the raw model for masked language modeling, but it's mostly intended to be fine-tuned on a downstream task.
+
+Note that this model is primarily aimed at being fine-tuned on tasks that use the whole sentence (potentially masked)
+to make decisions, such as sequence classification, token classification or question answering. For tasks such as text
+generation you should look at model like GPT2.
+
+### How to use
+
+You can use this model directly with a pipeline for masked language modeling:
+
+```python
+>>> from transformers import pipeline
+>>> unmasker = pipeline('fill-mask', model='flax-community/RoBERTa-large-finnish')
+>>> unmasker("Moikka olen <mask> kielimalli.")
+
+[{'sequence': 'Moikka olen uusi kielimalli.',
+  'score': 0.05129234120249748,
+  'token': 1825,
+  'token_str': ' uusi'},
+ {'sequence': 'Moikka olen toinen kielimalli.',
+  'score': 0.03112379088997841,
+  'token': 2194,
+  'token_str': ' toinen'},
+ {'sequence': 'Moikka olen myös kielimalli.',
+  'score': 0.025534993037581444,
+  'token': 491,
+  'token_str': ' myös'},
+ {'sequence': 'Moikka olen ensimmäinen kielimalli.',
+  'score': 0.020146571099758148,
+  'token': 2832,
+  'token_str': ' ensimmäinen'},
+ {'sequence': 'Moikka olen vapaa kielimalli.',
+  'score': 0.018089469522237778,
+  'token': 2257,
+  'token_str': ' vapaa'}]
+```
+
+Here is how to use this model to get the features of a given text in PyTorch:
+
+```python
+from transformers import RobertaTokenizer, RobertaModel
+tokenizer = RobertaTokenizer.from_pretrained('flax-community/RoBERTa-large-finnish')
+model = RobertaModel.from_pretrained('flax-community/RoBERTa-large-finnish')
+text = "Replace me by any text you'd like."
+encoded_input = tokenizer(text, return_tensors='pt')
+output = model(**encoded_input)
+```
+
+and in TensorFlow:
+
+```python
+from transformers import RobertaTokenizer, TFRobertaModel
+tokenizer = RobertaTokenizer.from_pretrained('flax-community/RoBERTa-large-finnish')
+model = TFRobertaModel.from_pretrained('flax-community/RoBERTa-large-finnish', from_pt=True)
+text = "Replace me by any text you'd like."
+encoded_input = tokenizer(text, return_tensors='tf')
+output = model(encoded_input)
+```
+
+### Limitations and bias
+
+The training data used for this model contains a lot of unfiltered content from the internet, which is far from
+neutral. Therefore, the model can have biased predictions.
+
+## Training data
+
+This Finnish RoBERTa model was pretrained on the combination of two datasets:
+- [mc4](https://huggingface.co/datasets/mc4), the dataset mC4 is a multilingual colossal, cleaned version of Common Crawl's web crawl corpus. We used the Finnish subset of the mC4 dataset
+- [Yle Finnish News Archive](http://urn.fi/urn:nbn:fi:lb-2017070501)
+
+Raw datasets were cleaned to filter out bad quality and non-Finnish examples. Together these cleaned datasets were around 51GB of text.
+
+## Training procedure
+
+### Preprocessing
+
+The texts are tokenized using a byte version of Byte-Pair Encoding (BPE) and a vocabulary size of 50265. The inputs of
+the model take pieces of 512 contiguous token that may span over documents. The beginning of a new document is marked
+with `<s>` and the end of one by `</s>`
+
+The details of the masking procedure for each sentence are the following:
+- 15% of the tokens are masked.
+- In 80% of the cases, the masked tokens are replaced by `<mask>`.
+
+- In 10% of the cases, the masked tokens are replaced by a random token (different) from the one they replace.
+- In the 10% remaining cases, the masked tokens are left as is.
+
+Contrary to BERT, the masking is done dynamically during pretraining (e.g., it changes at each epoch and is not fixed).
+
+### Pretraining
+
+The model was trained on TPUv3-8 VM, sponsored by the Hugging Face JAX/Flax community week event, for 2 epochs with a sequence length of 128 and continuing for one more epoch with a sequence length of 512. The optimizer used is Adafactor with a learning rate of 2e-4, \\(\beta_{1} = 0.9\\), \\(\beta_{2} = 0.98\\) and \\(\epsilon = 1e-6\\), learning rate warmup for 1500 steps and linear decay of the learning rate after.
+
+## Evaluation results
+
+Evaluation was done by fine-tuning the model on downstream text classification task with two different labeled datasets: [Yle News](https://github.com/spyysalo/yle-corpus) and [Eduskunta](https://github.com/aajanki/eduskunta-vkk). Yle News classification fine-tuning was done with two different sequence lengths: 128 and 512 but Eduskunta only with 128 sequence length.
+When fine-tuned on those datasets, this model (the first row of the table) achieves the following accuracy results compared to the [FinBERT (Finnish BERT)](https://huggingface.co/TurkuNLP/bert-base-finnish-cased-v1) and to our newer [Finnish RoBERTa-large](https://huggingface.co/Finnish-NLP/roberta-large-finnish) trained with larger dataset:
+
+|                                        | Average  | Yle News 128 length | Yle News 512 length | Eduskunta 128 length |
+|----------------------------------------|----------|---------------------|---------------------|----------------------|
+|flax-community/RoBERTa-large-finnish    |87.72     |94.42                |95.06                |73.67                 |
+|Finnish-NLP/roberta-large-finnish       |88.02     |94.53                |95.23                |74.30                 |
+|TurkuNLP/bert-base-finnish-cased-v1     |**88.82** |**94.90**            |**95.49**            |**76.07**             |
+
+To conclude, this model slightly loses to our newer [Finnish RoBERTa-large](https://huggingface.co/Finnish-NLP/roberta-large-finnish) model trained with larger dataset and also slightly loses to the [FinBERT (Finnish BERT)](https://huggingface.co/TurkuNLP/bert-base-finnish-cased-v1) model.
+
+## Team Members
+
+- Aapo Tanskanen, [Hugging Face profile](https://huggingface.co/aapot), [LinkedIn profile](https://www.linkedin.com/in/aapotanskanen/)
+- Rasmus Toivanen [Hugging Face profile](https://huggingface.co/RASMUS), [LinkedIn profile](https://www.linkedin.com/in/rasmustoivanen/)
+- Tommi Vehviläinen [Hugging Face profile](https://huggingface.co/Tommi)
+
+Feel free to contact us for more details 🤗