Update README.md

README.md

@@ -1,25 +1,23 @@
 ---
 language: fr
-license: mit
+license: apache-2.0
 tags:
 - legal
+- feature-extraction
 datasets: maastrichtlawtech/bsard
 pipeline_tag: fill-mask
 widget:
 - text: >-
     Chaque commune de la Région peut adopter un <mask> communal de
     développement, applicable à l'ensemble de son territoire.
+library_name: transformers
 ---
 
-# Legal-CamemBERT
+# Legal-DistilCamemBERT-base
 
-* Legal-DistilCamemBERT is a [DistilCamemBERT](https://huggingface.co/cmarkea/distilcamembert-base)-based model further pre-trained on [23,000+ statutory articles](https://huggingface.co/datasets/maastrichtlawtech/bsard) from the Belgian legislation.
-* We chose the following training set-up: 50k training steps (200 epochs) with batches of 32 sequences of length 512 with an initial learning rate of 5e-5.
-* Training was performed on one Tesla V100 GPU with 32 GB using the [code](https://github.com/huggingface/transformers/blob/main/examples/pytorch/language-modeling/run_mlm.py) provided by Hugging Face.
+This is a [DistilCamemBERT](https://huggingface.co/cmarkea/distilcamembert-base) model further pre-trained on 22,000+ legal articles from the Belgian legislation in French.
 
----
-
-### Load Pretrained Model
+## Usage
 
 ```python
 from transformers import AutoTokenizer, AutoModel
@@ -28,8 +26,32 @@ tokenizer = AutoTokenizer.from_pretrained("maastrichtlawtech/legal-distilcamembe
 model = AutoModel.from_pretrained("maastrichtlawtech/legal-distilcamembert")
 ```
 
-### About Us
+## Training
+
+#### Background
+
+We utilize the [distilcamembert-base](https://huggingface.co/cmarkea/distilcamembert-base) checkpoint and further pre-train it with a masked language modeling (MLM) objective on legislation in French using the [script](https://github.com/huggingface/transformers/blob/main/examples/pytorch/language-modeling/run_mlm.py) from Hugging Face.
+
+#### Hyperparameters
 
-The [Maastricht Law & Tech Lab](https://www.maastrichtuniversity.nl/about-um/faculties/law/research/law-and-tech-lab) develops algorithms, models, and systems that allow computers to process natural language texts from the legal domain.
+We train the model on a single Tesla V100 GPU with 32GBs of memory during 200 epochs (i.e., ~50k steps) using a batch size of 32. We use the AdamW optimizer with an initial learning rate of 5e-05, weight decay of 0.01, learning rate warmup over the first 500 steps, and linear decay of the learning rate. The sequence length was limited to 512 tokens.
 
-Author: [Antoine Louis](https://antoinelouis.co) on behalf of the [Maastricht Law & Tech Lab](https://www.maastrichtuniversity.nl/about-um/faculties/law/research/law-and-tech-lab).
+#### Data
+
+We use the [Belgian Statutory Article Retrieval Dataset (BSARD)](https://huggingface.co/datasets/maastrichtlawtech/bsard) to further pre-train the model. BSARD is a French native dataset for studying legal information retrieval, which consists of more than 22,600 statutory articles from Belgian law and about 1,100 legal questions posed by Belgian citizens and labeled by experienced jurists with relevant articles from the corpus.
+
+## Citation
+
+```bibtex
+@inproceedings{louis2023finding,
+  title = {Finding the Law: Enhancing Statutory Article Retrieval via Graph Neural Networks},
+  author = {Louis, Antoine and van Dijck, Gijs and Spanakis, Gerasimos},
+  booktitle = {Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics},
+  month = may,
+  year = {2023},
+  address = {Dubrovnik, Croatia},
+  publisher = {Association for Computational Linguistics},
+  url = {https://aclanthology.org/2023.eacl-main.203/},
+  pages = {2753–2768},
+}
+```