Hugging Face's logo

taln-ls2n
/
AdminBERT-4GB

Fill-Mask
Transformers
Safetensors
French
camembert
BERT
Administrative documents
Inference Endpoints
Model card Files Community
AdminBERT-4GB / README.md
TSebbag's picture
TSebbag
Update model spec
987f8ac verified
preview code
|
raw
history blame
681 Bytes
metadata
license: cc-by-nc-nd-4.0
datasets:
  - taln-ls2n/Adminset
language:
  - fr
library_name: transformers
tags:
  - camembert
  - BERT
  - Administrative documents

AdminBERT 4GB: A Small French Language model adapted to Administrative documents

AdminBERT-4GB is a French language model adapted on a large corpus of 10 millions French administrative texts. It is a derivative of CamemBERT model, which is based on the RoBERTa architecture. AdminBERT-4GB is trained using the Masked Language Modeling (MLM) objective with 30% mask rate for 2 epochs on 8 V100 GPUs. The dataset used for training is a sample of Adminset.