Create README.md

README.md

@@ -0,0 +1,25 @@
+---
+title: README
+emoji: 🏃
+colorFrom: gray
+colorTo: purple
+sdk: static
+pinned: false
+---
+
+# Model Description
+ClinicalDistilBERT was developed by training the [BioDistilBERT-cased](https://huggingface.co/nlpie/bio-distilbert-cased?text=The+goal+of+life+is+%5BMASK%5D.) model in a continual learning fashion for 3 epochs using a total batch size of 192 on the MIMIC-III notes dataset. 
+
+
+# Initialisation
+We initialise our model with the pre-trained checkpoints of the [BioDistilBERT-cased](https://huggingface.co/nlpie/bio-distilbert-cased?text=The+goal+of+life+is+%5BMASK%5D.) model available on Huggingface.
+
+# Architecture
+In this model, the size of the hidden dimension and the embedding layer are both set to 768. The vocabulary size is 28996. The number of transformer layers is 6 and the expansion rate of the feed-forward layer is 4. Overall, this model has around 65 million parameters.
+
+# Citation
+If you use this model, please consider citing the following paper:
+
+```bibtex
+
+```