thinhdeeptry123/vn-fin-ner-roberta

Financial NER for English Texts about the Vietnamese Economy

A roberta-base model fine-tuned for named entity recognition in English-language economic and financial texts about Vietnam. It tags six entity types: PER, TICKER, PERCENT, MONEY, LOC, ORG.

This model was built as a student project on a small, manually annotated dataset. Please read the Limitations section before using it.

Labels

Label	Meaning
ORG	Organizations, including government bodies and agencies (the original GOV label was merged into ORG)
LOC	Locations (countries, cities, provinces, ...)
PER	Person names
MONEY	Monetary amounts
PERCENT	Percentages
TICKER	Stock ticker symbols

Tags use the BIO scheme (13 classes: O plus B-/I- for each of the six labels). The LAW label of the original annotation is not supported (see Training data).

How to use

The model was trained on word-level tokens produced by a custom regex tokenizer that keeps numbers such as 1.5 and 2,000 intact. To get the same behavior, use the provided inference.py instead of the generic pipeline:

from huggingface_hub import hf_hub_download
import importlib.util

REPO = "thinhdeeptry123/vn-fin-ner-roberta"   
path = hf_hub_download(REPO, "inference.py")
spec = importlib.util.spec_from_file_location("inference", path)
inference = importlib.util.module_from_spec(spec); spec.loader.exec_module(inference)

tok, model = inference.load(REPO)
entities = inference.predict_entities(
    "The State Bank of Vietnam cut rates by 0.5% in Hanoi.", tok, model
)
print(entities)   # list of dicts with keys: text, label, start, end (character offsets)

Input should be English text. Inputs longer than about 256 subword tokens are truncated, so split long documents into sentences or short paragraphs first.

Training data

• Source: [FILL IN: where the texts come from and under what terms; the dataset is not released with this model].
• 800 English texts (from 604 source articles) annotated with 8 entity types, then cleaned automatically (no re-annotation).
• Cleaning decisions:
  • 15 sentences containing LAW (22 entities) were removed, because the label was too rare to train or evaluate on.
  • GOV was merged into ORG (89 spans), because the boundary between the two was ambiguous in the annotations.
  • 43 spans of the demonym "Vietnamese" were left untagged, because they had been labeled inconsistently (about half ORG, half LOC).
  • Span offsets were repaired: 70 spans expanded to whole words, 38 overlapping spans dropped (the longer span was kept), 1 empty span dropped.
• Final dataset: 785 sentences, 3,816 entities.

Label	Entities	Sentences
ORG	1,584	594
LOC	986	449
MONEY	500	215
PERCENT	260	165
TICKER	257	97
PER	229	169

Training procedure

• Base model: roberta-base, token classification head with 13 classes, label assigned to the first subword of each word.
• Hyperparameters (fixed in advance, not tuned on test folds): learning rate 3e-5, 10 epochs, batch size 16, AdamW with weight decay 0.01, linear schedule with 10% warmup, max length 256, cross-entropy loss, seed 42.
• Hardware: single Tesla T4 GPU.
• The published model was trained on all 785 cleaned sentences. Its performance is estimated from the cross-validation below; it has no separate held-out test set.

Evaluation

Protocol. 5-fold cross-validation using StratifiedGroupKFold: sentences from the same source article never appear in different folds, and rare labels are stratified across folds. Predictions from all folds are pooled before computing scores. Metric: strict entity-level F1 (exact span boundaries and label must match). 95% confidence intervals come from a document-level bootstrap (1,000 resamples). Systems are compared with a paired bootstrap on macro-F1.

Results for this model (RoBERTa, cross-entropy):

Label	Precision	Recall	F1 (95% CI)
PER	0.911	0.934	0.922 [0.882-0.956]
TICKER	0.835	0.883	0.858 [0.746-0.920]
PERCENT	0.909	0.919	0.914 [0.857-0.954]
MONEY	0.942	0.942	0.942 [0.919-0.965]
LOC	0.803	0.851	0.826 [0.800-0.851]
ORG	0.829	0.859	0.843 [0.823-0.864]

Macro-F1 0.884 [0.862-0.901], micro-F1 0.862 [0.847-0.876].

Comparison with other systems (same folds):

System	Macro-F1 (95% CI)	Micro-F1 (95% CI)
CRF with hand-crafted features	0.814 [0.787-0.833]	0.794 [0.777-0.810]
RoBERTa, cross-entropy (this model)	0.884 [0.862-0.901]	0.862 [0.847-0.876]
RoBERTa, weighted cross-entropy	0.883 [0.861-0.899]	0.862 [0.847-0.875]

RoBERTa is significantly better than the CRF in macro-F1 (difference 0.071, 95% CI [0.055, 0.088], p < 0.001). Weighted cross-entropy brings no significant change (difference -0.001, 95% CI [-0.009, +0.009], p = 0.836), so the simpler unweighted model was kept.

Seen vs. unseen entities. 56% of test entities (2,140 of 3,816) never appeared as an entity in the training folds. Recall is 0.849 on unseen and 0.915 on seen entities. The gap is largest for LOC (recall 0.62 on unseen locations vs. 0.935 on seen ones), so the model recognizes new place names much less reliably than ones it has seen.

Limitations

• Small dataset (785 sentences). Confidence intervals are wide, especially for TICKER, whose 257 entities are concentrated in only 97 sentences.
• Single training seed. The confidence intervals reflect sampling variation over documents, not variation from random initialization.
• Weakest labels are LOC and ORG (F1 0.826 and 0.843), even though they are the most frequent. [FILL IN after error analysis: main error types and likely causes.]
• No LAW label. Legal documents and regulations ("Decree No. ...", "Law on ...") are not recognized as entities.
• Government bodies are tagged as ORG; the model does not distinguish them from companies.
• "Vietnamese" and similar demonyms are deliberately untagged.
• Trained on text about Vietnam; performance on other domains, countries, or non-English text is untested and likely lower.
• Scores are not comparable with published NER benchmarks, since the dataset, label set, and strict evaluation differ.

Intended use and out-of-scope use

Intended for research, teaching, and prototyping of information extraction from English economic and financial text about Vietnam. It is not validated for automated decision-making, trading, compliance, or legal use. Outputs may contain errors and should be reviewed by a person.

Bias and privacy notes

The model tags person names (PER) and may therefore surface personal information present in its input. The training texts are not distributed with this model. [FILL IN: any licensing or copyright restrictions on the source texts.]

Citation and acknowledgements

• Base model: Liu et al. (2019), RoBERTa: A Robustly Optimized BERT Pretraining Approach, arXiv:1907.11692.
• Author: [FILL IN: name, institution, course/project].