Update README.md

README.md

@@ -1,3 +1,100 @@
 ---
 license: apache-2.0
+datasets:
+- wikipedia
+language:
+- it
+widget:
+- text: "milano è una [MASK] dell'italia"
+  example_title: "Example 1"
+- text: "il sole è una [MASK] della via lattea"
+  example_title: "Example 2"
+- text: "l'italia è una [MASK] dell'unione europea"
+  example_title: "Example 3"
 ---
+--------------------------------------------------------------------------------------------------
+
+<h1>Model: BLAZE 🔥</h1>
+<p>Language: IT<br>Version: 𝖬𝖪-𝖨</p>
+
+--------------------------------------------------------------------------------------------------
+
+<h3>Introduction</h3>
+
+This model is a <b>lightweight</b> and uncased version of <b>BERT</b> <b>[1]</b> for the <b>italian</b> language. With its <b>55M parameters</b> and <b>220MB</b> size, 
+it's <b>50% lighter</b> than a typical mono-lingual BERT model, and ideal 
+for circumstances where memory consumption and execution speed are critical aspects, while maintaining high quality results.
+
+
+<h3>Model description</h3>
+
+The model has been obtained by taking the multilingual <b>DistilBERT</b> <b>[2]</b> model (from the HuggingFace team: [distilbert-base-multilingual-cased](https://huggingface.co/distilbert-base-multilingual-cased)) as a starting point, 
+and then focusing it on the italian language while at the same time turning it into an uncased model by modifying the embedding layer 
+(as in <b>[3]</b>, but computing document-level frequencies over the <b>Wikipedia</b> dataset and setting a frequency threshold of 0.1%), which brings a considerable
+reduction in the number of parameters.
+
+In order to compensate for the deletion of cased tokens, which now forces the model to exploit lowercase representations of words which were previously capitalized, 
+the model has been further pre-trained on the italian split of the [Wikipedia](https://huggingface.co/datasets/wikipedia) dataset, using the <b>whole word masking [4]</b> technique to make it more robust 
+with respect to the new uncased representations.
+
+The resulting model has 55M parameters, a vocabulary of 13.832 tokens, and a size of 220MB, which makes it <b>50% lighter</b> than a typical mono-lingual BERT model and
+20% lighter than a typical mono-lingual DistilBERT model.
+
+
+<h3>Training procedure</h3>
+
+The model has been trained for <b>masked language modeling</b> on the italian <b>Wikipedia</b> (~3GB) dataset for 10K steps, using the AdamW optimizer, with a batch size of 512 
+(obtained through 128 gradient accumulation steps),
+a sequence length of 512, and a linearly decaying learning rate starting from 5e-5. The training has been performed using <b>dynamic masking</b> between epochs and
+exploiting the <b>whole word masking</b> technique.
+
+
+<h3>Performances</h3>
+
+The following metrics have been computed on the Part of Speech Tagging and Named Entity Recognition tasks, using the <b>UD Italian ISDT</b> and <b>WikiNER</b> datasets, respectively. 
+The PoST model has been trained for 5 epochs, and the NER model for 3 epochs, both with a constant learning rate, fixed at 1e-5. For Part of Speech Tagging, the metrics have been computed on the default test set
+provided with the dataset, while for Named Entity Recognition the metrics have been computed with a 5-fold cross-validation
+
+| Task | Recall | Precision | F1 |
+| ------ | ------ | ------ |  ------ |
+| Part of Speech Tagging | 97.48  | 97.29 | 97.37 |
+| Named Entity Recognition | 89.29  | 89.84 | 89.53 |
+
+The metrics have been computed at token level and macro-averaged over the classes.
+
+
+<h3>Demo</h3>
+
+You can try the model online (fine-tuned on named entity recognition) using this webapp:
+
+<h3>Quick usage</h3>
+
+```python
+from transformers import AutoTokenizer, AutoModel
+from transformers import pipeline
+
+tokenizer = AutoTokenizer.from_pretrained("osiria/blaze-it")
+model = AutoModel.from_pretrained("osiria/blaze-it")
+pipeline_mlm = pipeline(task="fill-mask", model=model, tokenizer=tokenizer)
+```
+
+
+<h3>Limitations</h3>
+
+This lightweight model is mainly trained on Wikipedia, so it's particularly suitable as an agile analyzer for large volumes of natively digital text taken 
+from the world wide web, written in a correct and fluent form (like wikis, web pages, news, etc.). It may show limitations when it comes to chaotic text, containing errors and slang expressions
+(like social media posts) or when it comes to domain-specific text (like medical, financial or legal content).
+
+<h3>References</h3>
+
+[1] https://arxiv.org/abs/1810.04805
+
+[2] https://arxiv.org/abs/1910.01108
+
+[3] https://arxiv.org/abs/2010.05609
+
+[4] https://arxiv.org/abs/1906.08101
+
+<h3>License</h3>
+
+The model is released under <b>Apache-2.0</b> license