viBERT-base

A Vietnamese RoBERTa-based language model pre-trained on CC-100 Vietnamese and custom Vietnamese corpus.

Model Description

viBERT-base is a BERT-base architecture model trained with RoBERTa-style pre-training on Vietnamese text data. It can be used for various Vietnamese NLP downstream tasks such as Named Entity Recognition, Text Classification, Question Answering, and more.

Model Architecture

Parameter	Value
Architecture	BERT-base
Hidden size	768
Attention heads	12
Hidden layers	12
Vocab size	41,035
Max sequence length	512
Parameters	~110M

Training Data

• CC-100 Vietnamese: Large-scale web crawl data
• Custom Vietnamese corpus: Additional curated Vietnamese text

Usage

Feature Extraction

from transformers import AutoModel, AutoTokenizer

# Load model and tokenizer
tokenizer = AutoTokenizer.from_pretrained("mainguyen9/viBERT-base")
model = AutoModel.from_pretrained("mainguyen9/viBERT-base")

# Encode text
text = "Xin chào Việt Nam"
inputs = tokenizer(text, return_tensors="pt")
outputs = model(**inputs)

# Get embeddings
last_hidden_state = outputs.last_hidden_state

Masked Language Modeling

from transformers import pipeline

fill_mask = pipeline("fill-mask", model="mainguyen9/viBERT-base")
result = fill_mask("Hà Nội là [MASK] đô của Việt Nam.")
print(result)

Fine-tuning for NER

from transformers import AutoModelForTokenClassification, AutoTokenizer

model = AutoModelForTokenClassification.from_pretrained(
    "mainguyen9/viBERT-base",
    num_labels=num_labels
)
tokenizer = AutoTokenizer.from_pretrained("mainguyen9/viBERT-base")

Benchmark Results

Task	Dataset	Metric	Score
NER	PhoNER_COVID19	F1	89.38
NLI	XNLI Vietnamese	Accuracy	71.06
Hate Speech	ViHSD	Accuracy	87.89

NER Performance Details (PhoNER_COVID19)

Fine-tuned with 5 epochs, batch size 32, learning rate 2e-5.

Entity Type	Precision	Recall	F1-Score	Support
AGE	90.91	97.27	93.98	586
DATE	98.20	99.17	98.68	3,026
GENDER	89.96	92.23	91.08	476
JOB	66.59	51.75	58.24	570
LOCATION	88.52	91.33	89.90	10,845
NAME	94.09	90.56	92.29	1,388
ORGANIZATION	77.02	78.05	77.53	1,640
PATIENT_ID	95.61	98.54	97.05	2,120
SYMPTOM_AND_DISEASE	82.84	74.70	78.56	2,158
TRANSPORTATION	85.63	91.41	88.43	489
Micro Average	89.09	89.69	89.38	23,298

NLI Performance (XNLI Vietnamese)

Fine-tuned with 5 epochs, batch size 64, learning rate 2e-5.

Metric	Score
Accuracy	71.06%
F1 (macro)	71.02%

Hate Speech Detection (ViHSD)

Fine-tuned with 5 epochs, batch size 8, learning rate 2e-5.

Class	Precision	Recall	F1-Score	Support
CLEAN	91.84%	96.40%	94.06%	5,548
OFFENSIVE	51.86%	40.77%	45.65%	444
HATE	67.32%	49.71%	57.19%	688
Accuracy			87.89%	6,680
Macro Avg	70.34%	62.29%	65.63%	6,680

Limitations

• Primarily trained on Vietnamese text; performance may vary for code-mixed text
• 512 token maximum sequence length

Citation

@misc{vibert-base,
  author = {Mai Nguyen},
  title = {viBERT-base: A Vietnamese RoBERTa Model},
  year = {2024},
  publisher = {Hugging Face},
  url = {https://huggingface.co/mainguyen9/viBERT-base}
}

License

MIT License