nishan-chatterjee/aspectbench-xlmr

AspectBench XLM-R

Model-only checkpoints for HBS and Slovenian document-level aspect-based sentiment analysis. This repository contains 4/4 language-mode checkpoint slots. It is used with the shared inference toolkit in nishan-chatterjee/aspect-based-sentiment-analysis.

What this repository contains

Each .pt file is a tensor-only state dictionary for a complete fine-tuned XLM-R encoder and its three-class classifier—not merely a small classification head. The language directories also include the configuration and tokenizer assets needed by the shared toolkit. Training data, optimizer state, cached features, logs, and row-level predictions are excluded.

The masked checkpoints use the standard XLM-R sequence-classification head. The unmasked checkpoints reproduce the paper's truncated-document classifier (XLM-R [CLS] representation → dropout → linear classifier). Because the architecture is selected by mode, this is intentionally not a generic Transformers save_pretrained() directory: use InferenceEngine or the aspectbench CLI below instead of calling AutoModel.from_pretrained() on this repository directly.

Input format

Every article must mark the target span with literal tags, even when using an unmasked checkpoint:

Tokom šestonedeljnog testiranja, redakcija je više puta kontaktirala <aspect>Primer Grupu</aspect> zbog nove usluge. Prvi odgovor <aspect>Primer Grupe</aspect> stigao je istog dana, a tehnički tim je zatim otklonio prijavljenu grešku bez dodatnih troškova. U završnom upitniku većina korisnika ocenila je podršku kao jasnu i pouzdanu.

• masked: the tagged text is replaced with [ASPECT]; the model does not see the target name.
• unmasked: the tags are removed and the model sees the target name.
• Gold sentiment is optional: -1 = negative, 0 = neutral, 1 = positive. It is reported in the result but never used to produce the prediction.

Available checkpoints

Language	Mode	Status	Best validation Macro-F1
hbs	masked	Available	0.9247
hbs	unmasked	Available (retrained)	0.9289
slovenian	masked	Available	0.8358
slovenian	unmasked	Available (retrained)	0.8336

availability.json contains the machine-readable selection record. A missing checkpoint is never replaced with a checkpoint from another mode or language.

Recovery provenance

The missing unmasked heads were retrained over all three fixed splits and selected only by validation Macro-F1. Optimizer state, logs, and row-level outputs are excluded from this model repository.

Language	Selected split	Validation Macro-F1	Mean test Macro-F1	Mean test QWK
hbs	0	0.9289	0.8014	0.7807
slovenian	2	0.8336	0.6818	0.6411

Getting started

Create the portable environment from the toolkit repository:

conda env create -f environment.yml
conda activate aspectbench

environment.yml is maintained once in the shared toolkit rather than copied into every model repository, preventing dependency versions from drifting between family releases.

Or install the runtime packages in an existing environment:

python -m pip install -U torch transformers accelerate huggingface-hub sentencepiece numpy spacy sentence-transformers

Download the toolkit and this model repository into the expected directory layout:

from pathlib import Path
from huggingface_hub import snapshot_download

ROOT = Path("huggingface")
snapshot_download(
    repo_id="nishan-chatterjee/aspect-based-sentiment-analysis",
    local_dir=ROOT,
)
snapshot_download(
    repo_id="nishan-chatterjee/aspectbench-xlmr",
    local_dir=ROOT / "models" / "xlmr",
)

The model repository includes the tokenizer and configuration assets required to reconstruct the architecture. No separate base-model cache is needed.

For a GitHub checkout, the equivalent one-command download and inference path is:

python huggingface/scripts/download.py --model xlmr
CUDA_VISIBLE_DEVICES=0 aspectbench infer --models xlmr --dataset hbs \
  --variant unmasked --input-doc 'Poziv za <aspect>Primer Grupu</aspect> je uspeo.' \
  --mc-passes 8

Python / Jupyter prediction

from pathlib import Path
import sys

ROOT = Path("huggingface").resolve()
sys.path.insert(0, str(ROOT / "scripts"))

from inference import InferenceEngine

engine = InferenceEngine(
    model_name="xlmr",
    language="hbs",
    mode="masked",
    model_root=ROOT / "models",
    device="cuda",  # use "cpu" when no GPU is available
)

prediction = engine.predict(
    {
        "article": "Tokom šestonedeljnog testiranja, redakcija je više puta kontaktirala <aspect>Primer Grupu</aspect> zbog nove usluge. Prvi odgovor <aspect>Primer Grupe</aspect> stigao je istog dana, a tehnički tim je zatim otklonio prijavljenu grešku bez dodatnih troškova. U završnom upitniku većina korisnika ocenila je podršku kao jasnu i pouzdanu.",
        "sentiment": 1,
    },
    mc_passes=10,
)
prediction

For a real batch, reuse the loaded engine:

records = [
    {"article": "Tokom šestonedeljnog testiranja, redakcija je više puta kontaktirala <aspect>Primer Grupu</aspect> zbog nove usluge. Prvi odgovor <aspect>Primer Grupe</aspect> stigao je istog dana, a tehnički tim je zatim otklonio prijavljenu grešku bez dodatnih troškova. U završnom upitniku većina korisnika ocenila je podršku kao jasnu i pouzdanu.", "sentiment": 1},
    {"article": "Pritužbe na <aspect>Drugi Sistem</aspect> nisu riješene.", "sentiment": -1},
]
predictions = engine.predict_batch(records, batch_size=2, mc_passes=10)

Command-line prediction

Run from the toolkit directory:

python scripts/predict.py \
  --model-name xlmr \
  --language hbs \
  --mode masked \
  --model-root models \
  --device cuda \
  --mc-passes 10 \
  --article 'Tokom šestonedeljnog testiranja, redakcija je više puta kontaktirala <aspect>Primer Grupu</aspect> zbog nove usluge. Prvi odgovor <aspect>Primer Grupe</aspect> stigao je istog dana, a tehnički tim je zatim otklonio prijavljenu grešku bez dodatnih troškova. U završnom upitniku većina korisnika ocenila je podršku kao jasnu i pouzdanu.' \
  --sentiment 1

Output fields

Field	Meaning
input_article	Original article, including <aspect> tags.
tagged_aspects	Target strings extracted from the tags.
aspect_used	Target representation actually supplied to the model.
gold_sentiment	Optional user-supplied reference label.
predicted_sentiment	Predicted integer label: -1, 0, or 1.
predicted_sentiment_name	Human-readable class name.
class_probabilities	Probability assigned to every sentiment class.
uncertainty_across_classes	Entropy, confidence, probability margin, and—when MC dropout is enabled—mutual information and vote statistics.
inference	Device, MC-dropout flag, and checkpoint path.

The .pt files contain model tensors only. Optimizer, scheduler, and gradient-scaler state is excluded.