mgroenendyk/bert-gov-canada-data-citation-classifier

BERT Government of Canada Data Citation Classifier

Binary sentence classifier for detecting citations to Government of Canada open datasets in academic and news text. Developed for the citation analysis reported in Groenendyk (2026).

Version 2. Supersedes v1. The v1 checkpoint was trained on a sentence-level split constructed without deduplication, so a minority of evaluation sentences also occurred in training and the reported metrics were not genuinely held out. This version was retrained on deduplicated partitions with no overlap between train, validation, and test.

Performance

Held-out test partition, n = 959 sentences (49.8% positive):

Metric	Score
Accuracy	0.975
Precision	0.973
Recall	0.977
F1	0.975
ROC-AUC	0.996

Baseline comparison

All approaches evaluated on the identical partition:

Approach	Accuracy	Precision	Recall	F1	ROC-AUC
Keyword matching	0.885	0.852	0.931	0.890	—
Enhanced heuristics	0.884	0.982	0.782	0.871	—
SPECTER embeddings + logistic regression	0.908	0.903	0.914	0.909	0.969
Fine-tuned BERT (this model)	0.975	0.973	0.977	0.975	0.996

A regular expression matching data, dataset, database, statistics, or microdata reaches 0.890 F1 on this partition. The fine-tuned model's margin over lexical matching is therefore modest — roughly eight and a half points of F1 — because the negative examples were drawn from the same documents as the positives and predominantly lack data vocabulary. Read the benchmark as characterising fit to this annotation scheme, not as evidence that the task requires a transformer.

Training data

• 6,514 labelled sentences: 3,257 positive, 3,257 negative
• Positives: 1,313 from academic literature (Scopus, Web of Science), 1,944 from Canadian news media (ProQuest Canadian Newsstream)
• Negatives: matched pairs — for each positive, one non-citing sentence from the same source document
• Manual annotation, Cohen's κ = 0.85 on a double-coded calibration subset
• After deduplication: 4,106 train / 813 validation / 959 test, no overlap

60% of positive examples derive from news media. Source genre was not retained as a field, so metrics cannot be stratified by genre.

Training configuration

Parameter	Value
Base model	bert-base-uncased
Epochs	2 (epoch-1 checkpoint retained on validation F1)
Learning rate	2e-5, AdamW, 10% warmup
Batch size	16
Max sequence length	128 WordPiece tokens
Padding	Dynamic, to batch maximum
Weight decay	0.01
Seed	42

Validation F1 was 0.964 after epoch 1 and 0.962 after epoch 2, with validation loss rising from 0.113 to 0.144 — the model converges within a single pass.

Limitations

Class balance. Partitions are 1:1 positive to negative. Real prevalence is under 3% of academic references to federal organisations. The implied false-positive rate is about 2.7%, so at a 3% base rate precision falls to roughly 0.53 — about half of positive predictions would be false. Deployment at scale requires pre-filtering candidate sentences or manual review of positives.

No provenance discrimination. Examination of all 24 test errors shows the model detects data vocabulary co-occurring with an institutional source, but has not learned whether the data is Government of Canada data. It flags citations to commercial datasets, United States government data, university research centre data, and journalist-obtained data. It also flags government publications — statistical compendia, studies, information inventories — rather than datasets.

Sequence length. Trained on single sentences at ≤128 tokens. Longer windows require sentence segmentation or an increased maximum length; a 100-word window commonly exceeds 128 tokens and would be truncated.

Jurisdiction and language. Depends on Canadian federal organisation names; retraining is required for other jurisdictions. English only — French-language citations are handled poorly.

Abbreviations. Underperforms on acronyms and short forms such as StatCan and ISC, which are underrepresented in training.

Usage

from transformers import pipeline

clf = pipeline(
    "text-classification",
    model="mgroenendyk/bert-gov-canada-data-citation-classifier",
)
clf("Data for this study were obtained from Fisheries and Oceans Canada.")

With explicit probabilities:

import torch
from transformers import AutoModelForSequenceClassification, AutoTokenizer

MODEL = "mgroenendyk/bert-gov-canada-data-citation-classifier"
tok = AutoTokenizer.from_pretrained(MODEL)
mdl = AutoModelForSequenceClassification.from_pretrained(MODEL).eval()

text = "According to Health Canada data, the findings show..."
enc = tok(text, return_tensors="pt", truncation=True, max_length=128)
with torch.no_grad():
    p = torch.softmax(mdl(**enc).logits, dim=-1)[0, 1].item()

print(f"citation probability: {p:.3f}")

Given the class-prevalence issue above, apply a higher threshold than 0.5 for unfiltered text, and verify positives manually.

Intended use

Research instrument for bibliometric analysis of government open data reuse. Not suitable for unreviewed production use: at realistic base rates, roughly half of positive predictions require manual confirmation.

Citation

@article{groenendyk2026citations,
  title   = {Citations of Government of Canada Open Data in Academic Literature:
             A Verified Corpus, a Fine-Tuned Detection Model, and the Limits of
             Dataset-Level Impact Measurement},
  author  = {Groenendyk, Michael},
  journal = {International Journal of Digital Curation},
  year    = {2026}
}

Resources

• Code and evaluation scripts: https://github.com/mikegroenendyk/bert-data-citation
• Verified citation corpus: Zenodo (DOI on publication)

Contact

Michael Groenendyk, Concordia University — michael.groenendyk@concordia.ca