Model Card for dmbert-lblod-dapt-1024

This model is a domain-adapted version of the Dutch DeBERTa-v3 model (yhavinga/dmbert-1024h-22l-fineweb2-2000000), specifically adapted to Flemish local administration documents using Masked Language Modeling (MLM).

Model Details

Model Description

• Developed by: [More Information Needed]
• Model type: DeBERTa-v3 (Masked Language Modeling)
• Language(s) (NLP): Dutch (specifically Flemish administrative/legal dialect)
• License: MIT
• Finetuned from model: yhavinga/dmbert-1024h-22l-fineweb2-2000000

Model Sources

• Repository: svercoutere/dmbert-lblod-dapt-1024
• Source Dataset: svercoutere/lblod-decisions-broad-corpus

---

Uses

Direct Use

This model can be directly used for Masked Language Modeling (fill-mask) to predict missing words in Flemish administrative texts. It is highly optimized to understand the syntactic structure, terminology, and legal phrasing of municipal decisions, college of mayor and aldermen (college van burgemeester en schepenen) acts, and official local government mandates.

Downstream Use

This domain-adapted backbone is ideal for further fine-tuning on downstream Flemish local government NLP tasks such as:

• Named Entity Recognition (NER) (e.g., detecting mandatees, locations, dates in decisions)
• Document Classification & Topic Labeling
• Information Extraction (e.g., extracting decision categories, article references)
• Semantic Search and Sentence Embeddings

---

Bias, Risks, and Limitations

• Formatting Bias: Because Flemish administrative documents heavily utilize numbered lists, articles, and section formatting, the raw model puts significant probability mass on digits, whitespace, and special formatting characters (e.g., §, 2, \n). Proper filtering or task-specific fine-tuning is recommended for text-only downstream applications.
• Geographic Bias: The training corpus represents Belgian-Flemish local governance terminology and structure. Performance on Netherlands Dutch administrative structures may vary.

---

How to Get Started with the Model

Use the code below to load and run inference with this model:

from transformers import pipeline

fill_mask = pipeline("fill-mask", model="svercoutere/dmbert-lblod-dapt-1024")
result = fill_mask("De gemeenteraad beslist om de<mask> goed te keuren.")
print(result)

---

Training Details

Training Data

• Dataset: svercoutere/lblod-decisions-broad-corpus packaged into svercoutere/dmbert-lblod-mlm-1024-sequences.
• Tokenization: Packed into continuous sequences of 1024 tokens.
• Masking: Standard MLM masking rate of 15%.

Training Hyperparameters

• Context Length: 1024 tokens
• Effective Batch Size: 32 (per-device batch size of 16 with gradient accumulation steps = 2)
• Optimizer: AdamW
• Learning Rate: 3e-5 with a Cosine learning rate scheduler
• Precision: FP16
• Epochs: 1 epoch

Environment & Speeds

• Hardware: 1 x NVIDIA A100-SXM4 (40GB VRAM)

---

Evaluation

Testing Data & Metrics

• Testing Data: A 5% validation split of the sequence-packed dataset.
• Metrics: Evaluation Loss & Perplexity (exponential of the loss).

Results

• Final Validation Loss: 0.3030
• Final Validation Perplexity: 1.35

---

Technical Specifications

• Model Architecture: DeBERTa-v3
• Parameters: ~230M parameters
• Layers: 22 layers
• Hidden Size: 1024