leminhhung0101/R-ViHSDModel

R-ViHSD — ViSoBERT + TF-IDF/LinearSVM Stacking

A Vietnamese hate-speech and text-noise classification pipeline combining:

• ViSoBERT for semantic representation.
• TF-IDF character + word n-grams for robust lexical features.
• LinearSVC models for hate-speech and noise classification.
• Logistic Regression stacking for the final hate-speech prediction.

The pipeline is trained using Out-of-Fold (OOF) stacking to reduce data leakage. After OOF training, the final base models are retrained on the full labeled dataset for inference on new data.

---

1. Task Definition

For each Vietnamese text sample, the system predicts two outputs.

Hate-Speech Label

CLEAN
OFFENSIVE
HATE

Noise Type

ORIGINAL
NO_DIACRITICS
TEENCODE
CHAR_REPEAT
PUNCT_NOISE
OBFUSCATION
MIXED

The minimum inference input format is:

id,text
0001,"sample text"
0002,"another sample"

The id column is preserved in the output but is not used as a model feature.

---

2. Overall Architecture

                            ┌─────────────────────────┐
                            │          TEXT           │
                            └────────────┬────────────┘
                                         │
                  ┌──────────────────────┼──────────────────────┐
                  │                      │                      │
                  ▼                      ▼                      ▼
         ┌────────────────┐    ┌──────────────────┐   ┌─────────────────┐
         │    ViSoBERT    │    │ TF-IDF char+word│   │ Surface Features│
         └───────┬────────┘    └────────┬─────────┘   └────────┬────────┘
                 │                      │                      │
       3 class probabilities      ┌─────┴─────┐                │
                                  │           │                │
                                  ▼           ▼                │
                           Hate LinearSVC  Noise LinearSVC     │
                                  │           │                │
                           3 decision     7 decision            │
                              scores         scores             │
                                  │           │                │
                                  │        softmax             │
                                  │           │                │
                                  └─────┬─────┘                │
                                        │                      │
                 ┌──────────────────────┴──────────────────────┘
                 │
                 ▼
       ┌──────────────────────┐
       │ 18 Stacking Features │
       └──────────┬───────────┘
                  │
                  ▼
       ┌──────────────────────┐
       │ Logistic Regression  │
       │     Meta-Model       │
       └──────────┬───────────┘
                  │
                  ▼
       CLEAN / OFFENSIVE / HATE

The final noise_type prediction does not pass through the meta-model. It is obtained directly from:

TF-IDF → Noise LinearSVC → argmax

---

3. Saved Model Structure

After training, the main working directory is expected to contain:

rvihsd_stacking_work/
│
├── visobert_full/
│   ├── config.json
│   ├── model.safetensors / pytorch_model.bin
│   ├── tokenizer_config.json
│   ├── tokenizer.json
│   └── ...
│
├── full_vectorizer.joblib
├── full_hate_svm.joblib
├── full_noise_svm.joblib
├── meta_model.joblib
│
├── visobert_oof.npy
├── svm_oof.npy
├── noise_oof_scores.npy
├── visobert_fold*_pred.npy
│
└── submissions/

Required Files for Inference

Only the following files are required for inference:

visobert_full/
full_vectorizer.joblib
full_hate_svm.joblib
full_noise_svm.joblib
meta_model.joblib

OOF files are used during training and evaluation but are not required for deployment.

---

4. ViSoBERT Model

Backbone:

uitnlp/visobert

ViSoBERT is fine-tuned for three-class hate-speech classification:

0 → CLEAN
1 → OFFENSIVE
2 → HATE

Main training configuration:

Parameter	Value
Backbone	uitnlp/visobert
Max sequence length	128
Epochs	3
Learning rate	2e-5
Weight decay	0.01
Warmup ratio	0.08
Label smoothing	0.05
Train batch size	128
Evaluation batch size	64
Gradient accumulation	2
Seed	42

During training, ViSoBERT uses weighted cross-entropy with sqrt-balanced class weights to reduce the effect of class imbalance.

The model produces three probabilities:

P(CLEAN)
P(OFFENSIVE)
P(HATE)

These three values are passed to the stacking meta-model.

---

5. TF-IDF Features

The pipeline combines two TF-IDF vectorizers.

Character TF-IDF

analyzer="char"
ngram_range=(3, 5)
max_features=150_000
min_df=2
sublinear_tf=True

Word TF-IDF

analyzer="word"
ngram_range=(1, 2)
max_features=80_000
min_df=2
sublinear_tf=True

The two sparse matrices are concatenated:

Character TF-IDF + Word TF-IDF
              │
              ▼
       Sparse Feature Matrix

Character n-grams are particularly useful for noisy social-media text, including:

• teencode,
• misspellings,
• missing Vietnamese diacritics,
• repeated characters,
• obfuscation,
• punctuation noise.

---

6. LinearSVC Models

Two independent LinearSVC models are used.

Hate-Speech SVM

Classes:

CLEAN
OFFENSIVE
HATE

Configuration:

LinearSVC(
    C=2.0,
    class_weight="balanced",
    random_state=42
)

The stacking model uses the three decision_function scores, not calibrated probabilities.

Noise SVM

Classes:

ORIGINAL
NO_DIACRITICS
TEENCODE
CHAR_REPEAT
PUNCT_NOISE
OBFUSCATION
MIXED

Configuration:

LinearSVC(
    C=2.0,
    class_weight="balanced",
    random_state=42
)

For stacking, the seven decision scores are transformed using:

softmax(noise_score, axis=1)

These values are only used as stacking features and should not be interpreted as calibrated probabilities.

---

7. Surface Features

The pipeline also extracts five handcrafted features:

1. log(1 + text length)
2. punctuation ratio
3. Vietnamese-diacritic ratio
4. repeated-character score
5. obfuscation score

The final meta-model input consists of:

ViSoBERT probabilities        3
Hate SVM decision scores      3
Noise SVM softmax scores      7
Surface features              5
-------------------------------
Total                        18

The feature order must remain exactly the same during inference.

---

8. Meta-Model

The final hate-speech meta-model is:

StandardScaler
      ↓
LogisticRegression

Configuration:

Pipeline([
    ("scale", StandardScaler()),
    ("lr", LogisticRegression(
        C=1.0,
        max_iter=3000,
        class_weight="balanced",
        random_state=42
    ))
])

Input:

18 features

Output:

CLEAN / OFFENSIVE / HATE

The model is saved as:

meta_model.joblib

---

9. Out-of-Fold Stacking

The meta-model should not be trained on predictions produced by base models that have already seen the same samples.

The training pipeline therefore uses:

StratifiedGroupKFold(
    n_splits=5,
    shuffle=True,
    random_state=42
)

Workflow:

Fold 1 → train base models on other folds → predict Fold 1
Fold 2 → train base models on other folds → predict Fold 2
...
Fold 5 → train base models on other folds → predict Fold 5

These predictions form the Out-of-Fold feature matrix.

The meta-model is then trained on:

ViSoBERT OOF predictions
+
Hate SVM OOF scores
+
Noise SVM OOF scores
+
Surface features
        ↓
Logistic Regression Meta-Model

Grouping based on normalized text can also be used to reduce duplicate or augmentation leakage across folds.

---

10. Final Training

After OOF features are generated and the meta-model is trained:

1. The TF-IDF vectorizer is fitted again on all labeled data.
2. The hate-speech LinearSVC is trained on all labeled data.
3. The noise LinearSVC is trained on all labeled data.
4. ViSoBERT is fine-tuned on all labeled data.
5. All final models are saved for inference.

If:

USE_VALIDATION_FOR_FINAL_TRAIN = True

the final models use:

training_set + validation_set

---

11. Environment Requirements

Recommended installation:

pip install -U \
    "transformers>=4.46" \
    "accelerate>=1.0" \
    "scikit-learn>=1.4" \
    sentencepiece \
    joblib \
    scipy \
    pandas \
    numpy \
    torch

Main dependencies:

Python
PyTorch
Transformers
scikit-learn
SciPy
NumPy
Pandas
Joblib
SentencePiece

CUDA GPU support is recommended for ViSoBERT inference but is not required.

---

12. Loading the Models

Example:

from pathlib import Path
import joblib

from transformers import (
    AutoTokenizer,
    AutoModelForSequenceClassification,
)

WORK_DIR = Path(
    "/content/drive/MyDrive/rvihsd_stacking_work"
)

tokenizer = AutoTokenizer.from_pretrained(
    WORK_DIR / "visobert_full"
)

visobert = AutoModelForSequenceClassification.from_pretrained(
    WORK_DIR / "visobert_full"
)

vectorizer = joblib.load(
    WORK_DIR / "full_vectorizer.joblib"
)

hate_svm = joblib.load(
    WORK_DIR / "full_hate_svm.joblib"
)

noise_svm = joblib.load(
    WORK_DIR / "full_noise_svm.joblib"
)

meta_model = joblib.load(
    WORK_DIR / "meta_model.joblib"
)

full_vectorizer.joblib contains a custom DualTfidf class. The inference environment must define a compatible DualTfidf class before loading the file with joblib.

---

13. Inference Flow

For each new text sample:

text
 │
 ├── ViSoBERT
 │      └── 3 class probabilities
 │
 ├── TF-IDF
 │      ├── Hate LinearSVC
 │      │      └── 3 decision scores
 │      │
 │      └── Noise LinearSVC
 │             ├── 7 decision scores
 │             └── softmax → 7 stacking features
 │
 └── Surface features
        └── 5 features

The final stacking input is:

meta_X = np.hstack([
    visobert_prob,      # 3
    hate_svm_score,     # 3
    noise_soft,         # 7
    surface_features,   # 5
])

The feature dimension must satisfy:

assert meta_X.shape[1] == 18

Final hate-speech prediction:

hate_pred = meta_model.predict(meta_X)

Final noise prediction:

noise_pred = noise_score.argmax(axis=1)

---

14. Input Format

A new CSV file should contain at least:

id,text
1,"first sentence"
2,"second sentence"
3,"third sentence"

Additional columns may exist, but inference should only depend on:

id
text

This prevents accidental use of labels or unrelated metadata.

---

15. Output Format

Recommended output:

id,pred_label,pred_noise_type
1,CLEAN,ORIGINAL
2,OFFENSIVE,TEENCODE
3,HATE,NO_DIACRITICS

Columns:

Column	Description
id	Original sample ID
pred_label	CLEAN, OFFENSIVE, or HATE
pred_noise_type	One of the seven supported noise classes

---

16. Label Mapping

Hate-Speech Labels

LABELS = [
    "CLEAN",
    "OFFENSIVE",
    "HATE",
]

Mapping:

0 → CLEAN
1 → OFFENSIVE
2 → HATE

Noise Labels

NOISE_LABELS = [
    "ORIGINAL",
    "NO_DIACRITICS",
    "TEENCODE",
    "CHAR_REPEAT",
    "PUNCT_NOISE",
    "OBFUSCATION",
    "MIXED",
]

Mapping:

0 → ORIGINAL
1 → NO_DIACRITICS
2 → TEENCODE
3 → CHAR_REPEAT
4 → PUNCT_NOISE
5 → OBFUSCATION
6 → MIXED

Do not change the class order when using the already-trained models.

---

17. Recommended Inference Checks

Useful safety checks:

assert len(output) == len(test_df)
assert output["id"].is_unique

assert set(
    output["pred_label"]
).issubset(LABELS)

assert set(
    output["pred_noise_type"]
).issubset(NOISE_LABELS)

assert meta_X.shape[1] == 18

If the stacking matrix does not contain exactly 18 features, the inference feature construction no longer matches training.

---

18. Components That Must Stay Consistent

When using the existing trained models, keep the following unchanged:

• hate-speech label order,
• noise label order,
• MAX_LENGTH = 128,
• tokenizer saved in visobert_full,
• meta_surface_features implementation,
• DualTfidf implementation,
• 18-feature stacking order,
• noise-score softmax transformation before the meta-model.

The stacking order must remain:

[ViSoBERT: 3]
+
[Hate SVM: 3]
+
[Noise SVM: 7]
+
[Surface Features: 5]

---

19. Do New Test Sets Require Retraining?

No.

If the following trained artifacts are available:

visobert_full/
full_vectorizer.joblib
full_hate_svm.joblib
full_noise_svm.joblib
meta_model.joblib

a new dataset only requires:

LOAD MODELS
     ↓
LOAD NEW CSV
     ↓
TF-IDF + SVM INFERENCE
     ↓
ViSoBERT INFERENCE
     ↓
SURFACE FEATURE EXTRACTION
     ↓
STACKING
     ↓
SAVE PREDICTIONS

There is no need to rerun:

5-fold OOF training
TF-IDF fitting
SVM training
ViSoBERT fine-tuning
Meta-model training

---

20. Training Cache

The notebook may use:

REUSE_CACHE = True

Typical cache files include:

svm_oof.npy
noise_oof_scores.npy
visobert_oof.npy
visobert_fold*_pred.npy

These files are useful for resuming training or reusing OOF predictions.

They are not required for deployment.

---

21. Main Training Hyperparameters

N_FOLDS = 5
SEED = 42

MODEL_NAME = "uitnlp/visobert"
MAX_LENGTH = 128
EPOCHS = 3

TRAIN_BATCH_SIZE = 128
EVAL_BATCH_SIZE = 64
GRAD_ACCUM_STEPS = 2

LEARNING_RATE = 2e-5
WEIGHT_DECAY = 0.01
WARMUP_RATIO = 0.08
LABEL_SMOOTHING = 0.05

CHAR_NGRAM = (3, 5)
WORD_NGRAM = (1, 2)

CHAR_MAX_FEATURES = 150_000
WORD_MAX_FEATURES = 80_000

MIN_DF = 2

SVM_C_HATE = 2.0
SVM_C_NOISE = 2.0

META_C = 1.0

---

22. Evaluation Metric

Both tasks are evaluated using Macro-F1.

Hate-speech classification:

f1_score(
    y_hate,
    hate_pred,
    average="macro"
)

Noise classification:

f1_score(
    y_noise,
    noise_pred,
    average="macro"
)

The notebook may also compute a combined score:

0.85 × Hate Macro-F1
+
0.15 × Noise Macro-F1

This README intentionally does not report a fixed F1 score because the actual metric depends on the specific training run and cached predictions.

---

23. Minimal Deployment Package

For deployment on another machine, the project can be organized as:

model/
├── visobert_full/
├── full_vectorizer.joblib
├── full_hate_svm.joblib
├── full_noise_svm.joblib
├── meta_model.joblib
├── inference.py
└── README.md

A command-line inference interface may look like:

python inference.py \
    --input new_test.csv \
    --output predictions.csv \
    --model-dir model

---

24. Notes

• The meta-model predicts only the hate-speech label.
• The noise label is produced directly by the noise LinearSVC.
• LinearSVC.decision_function() values are not probabilities.
• The softmax applied to noise scores is used as a stacking transformation rather than probability calibration.
• Model compatibility depends on preserving the preprocessing and feature-ordering logic used during training.
• When transferring joblib artifacts between environments, compatible versions of Python and scikit-learn are recommended.

---

25. Summary

The final inference system is:

ViSoBERT
   +
TF-IDF Character/Word Features
   +
Hate LinearSVC
   +
Noise LinearSVC
   +
Surface Features
   ↓
Logistic Regression Stacking
   ↓
Final Hate-Speech Prediction

Noise LinearSVC
   ↓
Final Noise-Type Prediction

This architecture combines transformer-based semantic information with sparse lexical features that are robust to noisy Vietnamese social-media text.