sifat6412/USENIX25-Security-Issue-Classifier

USENIX25 — RoBERTa Security Issue Classifier

This repository hosts a Hugging Face copy of the fine-tuned RoBERTa security-issue classifier associated with the USENIX Security 2025 paper “I wasn't sure if this is indeed a security risk”: Data-driven Understanding of Security Issue Reporting in GitHub Repositories of Open Source npm Packages by Rajdeep Ghosh, Shiladitya De, and Mainack Mondal.

Paper: https://arxiv.org/abs/2506.07728
Original research artifact: https://doi.org/10.5281/zenodo.15614029
Hosted checkpoint: https://huggingface.co/sifat6412/USENIX25-Security-Issue-Classifier

Attribution: The model and research are the work of the original paper authors. This repository is a hosted copy, not a claim of original model authorship or independent reproduction of the paper's results. Verify permission to redistribute the weights and the applicable license before making this repository public. A generic license: cc value is not sufficient to establish the checkpoint's license.

Model Details

Model Description

The classifier predicts whether a GitHub issue report is security-related based on its title and body. It is a binary sequence-classification model, not a vulnerability verifier.

• Developed by: Rajdeep Ghosh, Shiladitya De, and Mainack Mondal (original research)
• Shared by: sifat6412 (Hugging Face hosting)
• Model type: Fine-tuned RoBERTa-base sequence classifier, two classes
• Language: English issue reports (study context)
• License: Not verified for redistribution; consult original artifact terms
• Fine-tuned from: FacebookAI/roberta-base

Model Sources

• Paper: https://arxiv.org/abs/2506.07728
• Publication: 34th USENIX Security Symposium (2025)
• Research artifact: https://doi.org/10.5281/zenodo.15614029
• Model repository: https://huggingface.co/sifat6412/USENIX25-Security-Issue-Classifier

Uses

Direct Use

Screen GitHub issue titles and descriptions to prioritize issues potentially related to software security. The intended prediction is security-related issue versus non-security-related issue.

Downstream Use

Research on untagged security reports, issue triage, open-source security reporting, and comparative evaluations on additional repositories or ecosystems.

Bias, Risks, and Limitations

• The original work studied GitHub issue reports from repositories of npm packages; transfer to other ecosystems has not been established by the reported evaluation.
• Issue reports may be incomplete, ambiguous, or contain external links instead of details.
• Security issues without characteristic vocabulary may be missed; unrelated issues with security-related terms may be flagged.
• A 512-token inference cutoff can omit information from long issues.
• Scores are model probabilities, not independently calibrated estimates of real-world vulnerability prevalence.

Recommendations

Use predictions for triage and empirical analysis, check the label mapping in config.json, evaluate on the target repository and class distribution, and manually review consequential cases.

How to Get Started with the Model

Install dependencies:

pip install torch transformers

import torch
from transformers import AutoTokenizer, AutoModelForSequenceClassification

model_id = "sifat6412/USENIX25-Security-Issue-Classifier"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForSequenceClassification.from_pretrained(model_id)
model.eval()

title = "Authentication bypass in API route"
body = "An unauthenticated request can access a restricted endpoint."
text = f"Issue title: {title} Issue details: {body}"

inputs = tokenizer(text, return_tensors="pt", truncation=True,
                   max_length=512)
with torch.no_grad():
    probabilities = torch.softmax(model(**inputs).logits, dim=-1)[0]

# Original artifact's inference code maps index 1 to Security Issue.
# Confirm this against config.json for the uploaded checkpoint.
print("Security-related probability:", probabilities[1].item())
print("Predicted security-related:", probabilities[1].item() >= 0.5)

Training Details

Training Data

The authors manually labeled 2,100 GitHub issues, including 1,042 security-related and 1,058 non-security-related reports. The paper's larger collection covered 10,907,467 issues across GitHub repositories associated with 45,466 npm packages; this full collection is not the labeled training set.

Training Procedure

The paper formulated security-issue identification as supervised binary text classification using the issue title and description. It compared multiple architectures and selected fine-tuned RoBERTa for subsequent analysis. The authors report systematic hyperparameter optimization using Optuna and an 80%/20% training/validation split.

Preprocessing

At inference, the accompanying artifact uses a single text string containing the issue title and details. The example above preserves this format, with max_length=512 as used by the supplied evaluation script. This card does not claim the paper specified every tokenizer or preprocessing option.

Training Hyperparameters

The exact learning rate, batch size, epochs, numerical precision, and optimizer settings for this hosted checkpoint are not independently verified here. Consult the original code and configuration in the research artifact rather than assuming defaults.

Speeds, Sizes, Times

Not independently benchmarked for this hosted checkpoint; inference times depend on device, batch size, and input length.

Evaluation

Testing Data, Factors & Metrics

Testing Data

The paper reports results on its held-out split of manually annotated issue reports, as well as a separate manual review of 100 previously unseen issues.

Factors

The reported binary evaluation compares security-related and non-security-related issue classes.

Metrics

Accuracy, macro F1, ROC-AUC, class-wise precision and recall, and an additional manual-check confusion matrix.

Results

Reported by the original authors (not rerun for this Hugging Face upload):

Metric	Result
Accuracy	0.94
Macro F1	0.94
ROC-AUC	0.94
Security-related precision	0.95
Security-related recall	0.93
Non-security precision	0.93
Non-security recall	0.95

The independently manually checked 100-issue sample had 96 correct classifications (6 true security positives, 90 true non-security negatives, 3 missed security issues, and 1 false positive).

Summary

The authors applied their classifier to untagged npm-repository issues and identified 1,617,738 potentially security-related untagged issues.

Model Examination

The paper reports examining influential tokens using final-layer RoBERTa attention scores. See the paper's appendix for the authors' analysis; attention scores alone should not be treated as causal explanations of predictions.

Environmental Impact

Training hardware, cloud region, duration, and carbon emissions for the uploaded checkpoint have not been independently verified. No numerical carbon estimate is claimed.

Technical Specifications

Model Architecture and Objective

RoBERTa-base with a two-class sequence-classification head. Input consists of issue title and description, and output is a security-relevance class score. Check the checkpoint's config.json for the exact id2label mapping.

Compute Infrastructure

Hardware

Not confirmed from the hosted checkpoint.

Software

Compatible with Hugging Face transformers and PyTorch. Original experiments used Optuna for model-selection hyperparameter tuning; precise package versions should be checked in the original artifact.

Citation

Paper:

Ghosh, Rajdeep, Shiladitya De, and Mainack Mondal. "" I wasn't sure if this is indeed a security risk": Data-driven Understanding of Security Issue Reporting in {GitHub} Repositories of Open Source npm Packages." 34th USENIX Security Symposium (USENIX Security 25). 2025.

BibTeX:

@inproceedings{ghosh2025wasn,
  title={" I wasn't sure if this is indeed a security risk": Data-driven Understanding of Security Issue Reporting in $\{$GitHub$\}$ Repositories of Open Source npm Packages},
  author={Ghosh, Rajdeep and De, Shiladitya and Mondal, Mainack},
  booktitle={34th USENIX Security Symposium (USENIX Security 25)},
  pages={2145--2164},
  year={2025}
}

Glossary

• Security-related issue: A GitHub issue report classified as concerning a security problem; this does not imply a verified vulnerability.
• False negative: A genuinely security-related issue incorrectly predicted to be non-security-related.

More Information

See the original paper, research artifact, and hosted checkpoint.

Model Card Authors

Prepared for the sifat6412/USENIX25-Security-Issue-Classifier hosting repository, based on the original authors' published research; not an official model card from the paper authors.

Model Card Contact

For original model/research questions, consult the publication and associated artifact. For hosting issues, contact the maintainer through the Hugging Face repository.