linhanhvlog123/mbert-viquad2.0-qa

🇻🇳 mBERT fine-tuned on UIT-ViQuAD 2.0 for Vietnamese Question Answering

This repository provides a multilingual BERT (mBERT) model fine-tuned for extractive Question Answering (QA) on UIT-ViQuAD 2.0, a Vietnamese Machine Reading Comprehension (MRC) benchmark that includes both answerable and unanswerable questions, following the SQuAD 2.0 setting.

The model is trained using the Hugging Face run_qa.py pipeline with a fixed hyperparameter configuration, enabling a fair and controlled comparison with other multilingual QA models such as XLM-RoBERTa.

---

📌 Task Description

• Task: Extractive Question Answering (Machine Reading Comprehension)

• Language: Vietnamese

• Input:

  • A Vietnamese context paragraph
  • A question related to the context

• Output:

  • An extracted answer span from the context or
  • An empty string if the question is unanswerable

This setup strictly follows the SQuAD 2.0 paradigm, where the model must:

1. Predict correct answer spans
2. Detect questions with no valid answer in the given context

---

📚 Dataset: UIT-ViQuAD 2.0

UIT-ViQuAD 2.0 is a large-scale Vietnamese QA benchmark released for the VLSP 2021 Machine Reading Comprehension shared task, designed to address the lack of Vietnamese datasets containing unanswerable questions.

Dataset statistics

Split	# Questions
Train	28,457
Dev	~5,700
Public Test	~3,821
Private Test	3,712

• Text source: Wikipedia-style Vietnamese articles
• Domains: history, geography, culture, science, etc.
• Annotation: human-annotated answer spans and unanswerable labels

---

🧾 Data Format

1️⃣ SQuAD-style format (for evaluation)

{
  "data": [
    {
      "title": "...",
      "paragraphs": [
        {
          "context": "...",
          "qas": [
            {
              "id": "uit_000001",
              "question": "...",
              "answers": [
                {
                  "text": "...",
                  "answer_start": 123
                }
              ],
              "is_impossible": false
            }
          ]
        }
      ]
    }
  ]
}

This format is required by the official SQuAD v2.0 evaluation script (evaluate-v2.0.py) to compute Exact Match (EM) and F1-score.

---

2️⃣ Hugging Face QA format (for training & inference)

To train with Hugging Face run_qa.py, the dataset is normalized into the following flat QA format:

{
  "id": "uit_000001",
  "title": "...",
  "context": "...",
  "question": "...",
  "answers": {
    "text": ["..."],
    "answer_start": [123]
  }
}

Unanswerable questions are represented with empty text and answer_start fields.

---

🔄 Data Preprocessing Pipeline

The dataset is preprocessed in two stages to ensure compatibility with both training and evaluation tools.

🔹 Stage 1: UIT-ViQuAD → SQuAD format

Purpose:

• Preserve hierarchical structure (paragraphs, qas)
• Enable evaluation using the official Stanford SQuAD v2.0 script

🔹 Stage 2: SQuAD format → Hugging Face QA format

Purpose:

• Enable training and inference with run_qa.py

Key steps:

• Flatten paragraph-level data
• Normalize answer spans
• Retain unanswerable question labels
• Validate span offsets

---

🧠 Model

• Base model: bert-base-multilingual-cased
• Architecture: Transformer encoder (mBERT)
• Head: Span-based QA head (start/end logits)
• Tokenizer: Multilingual BERT tokenizer

---

⚙️ Training Configuration

The model is trained using a shared hyperparameter configuration, identical to other baseline models in this project.

Model:              bert-base-multilingual-cased
Max sequence length: 512
Document stride:     256
Train batch size:    16
Eval batch size:     8
Learning rate:       2e-5
Epochs:              3
Optimizer:           AdamW
FP16:                Enabled
Seed:                42
Version 2 QA:        Enabled (unanswerable questions)

Training is performed using the Hugging Face Trainer API via run_qa.py.

---

📊 Evaluation Results (Private Test Set)

Evaluation is conducted using the official SQuAD v2.0 evaluation script on the private test set.

{
  "exact": 49.33,
  "f1": 60.36,
  "HasAns_exact": 41.64,
  "HasAns_f1": 57.41,
  "NoAns_exact": 67.20,
  "NoAns_f1": 67.20,
  "total": 3712
}

Observations

• mBERT shows strong performance on No-Answer detection
• Higher overall EM and F1 compared to XLM-R baseline
• Performance gap suggests different inductive biases between multilingual pre-trained models

---

🚀 Usage

Load the model

from transformers import AutoModelForQuestionAnswering, AutoTokenizer

model = AutoModelForQuestionAnswering.from_pretrained(
    "linhanhvlog123/mbert-viquad2.0-qa"
)
tokenizer = AutoTokenizer.from_pretrained(
    "linhanhvlog123/mbert-viquad2.0-qa"
)

Inference example

from transformers import pipeline

qa = pipeline("question-answering", model=model, tokenizer=tokenizer)

qa({
    "context": "...",
    "question": "..."
})

---

🏁 Notes

• This model serves as a baseline multilingual QA system for Vietnamese.

• All hyperparameters are kept fixed to ensure fair comparison across models.

• Further improvements may be achieved via:

  • Model-specific hyperparameter tuning
  • Larger batch sizes
  • Additional Vietnamese pretraining

---

📜 Citation

If you use UIT-ViQuAD 2.0 or this model, please cite the original dataset paper from VLSP 2021.