dilekmandal/owasp-java-security-qwen2.5-coder-7b-lora

OWASP Java Security — Qwen2.5-Coder-7B LoRA

Model Summary

This is an experimental LoRA fine-tuned adapter based on Qwen2.5-Coder-7B-Instruct for Java source-code vulnerability classification. It is trained on a dataset derived from OWASP Benchmark Java.

The model is designed to classify Java code samples as vulnerable or not vulnerable and to produce structured JSON containing vulnerability status, CWE information, and category information.

This model is intended for research and experimentation. It is not a replacement for professional security review, static analysis, or dynamic application security testing.

Model Details

Model Description

• Developed by: Dilek Mandal
• Shared by: Dilek Mandal
• Model type: LoRA adapter for causal language modeling
• Language(s): Java source code; English-language instructions
• Fine-tuned from: Qwen/Qwen2.5-Coder-7B-Instruct
• Library: Hugging Face Transformers and PEFT
• Model repository: dilekmandal/owasp-java-security-qwen2.5-coder-7b-lora
• Training dataset: dilekmandal/owasp-benchmark-java-security

The repository contains a LoRA adapter rather than a standalone copy of the complete base model. The base model is required for inference.

Intended Uses

Direct Use

The adapter can be loaded with its base model to experiment with Java vulnerability classification. The intended output is a structured JSON response identifying whether a sample is vulnerable and, where applicable, its CWE and category.

Downstream Use

Potential applications include:

• Research into machine-learning-based software vulnerability detection.
• Educational demonstrations of Java security weaknesses.
• Preliminary security triage of Java source code.
• Comparisons of code-security fine-tuning approaches.

Any integration into a development or security workflow should include independent validation and human review.

Out-of-Scope Use

This model should not be used:

• As the sole mechanism for approving code as secure.
• As a replacement for security professionals or established security testing tools.
• To make claims of production-level vulnerability detection without independent evaluation.
• As evidence that code is free from vulnerabilities when the model predicts that it is not vulnerable.

Bias, Risks, and Limitations

The model learns patterns from its training data and may fail to generalize to code that differs from the benchmark examples.

Potential limitations include:

• False negatives: vulnerabilities may be missed.
• False positives: safe code may be classified as vulnerable.
• Incorrect CWE or category predictions.
• Sensitivity to code formatting, context, and prompt wording.
• Limited evidence of performance on real-world applications.

Dataset Leakage and Evaluation Limitations

Dataset auditing identified exact normalized-code overlaps and shared code families across training, validation, and test splits. As a result, the evaluation may be optimistic because related examples can occur in more than one split.

The reported evaluation results should therefore be interpreted as preliminary benchmark results, not proof of generalization to unseen software.

A more reliable estimate requires deduplication, family-based splitting, and evaluation on an independent dataset.

Recommendations

Users should:

• Review predictions before taking security-related action.
• Combine model outputs with static analysis, dynamic testing, and manual review.
• Evaluate the model on independently collected code before operational deployment.
• Avoid interpreting a non-vulnerable prediction as proof that code is secure.

How to Get Started

Install the required packages:

pip install torch transformers peft accelerate

Load the base model and adapter:

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

base_model_id = "Qwen/Qwen2.5-Coder-7B-Instruct"
adapter_id = "dilekmandal/owasp-java-security-qwen2.5-coder-7b-lora"

tokenizer = AutoTokenizer.from_pretrained(base_model_id)

base_model = AutoModelForCausalLM.from_pretrained(
    base_model_id,
    torch_dtype="auto",
    device_map="auto"
)

model = PeftModel.from_pretrained(base_model, adapter_id)
model.eval()

java_code = """
public class Example {
    public String query(String input) {
        return "SELECT * FROM users WHERE name = '" + input + "'";
    }
}
"""

messages = [
    {
        "role": "system",
        "content": (
            "You are a Java application security analyzer. "
            "Return only JSON with vulnerable, cwe, and category."
        )
    },
    {
        "role": "user",
        "content": (
            "Analyze the following Java source code:\n"
            f"```java\n{java_code}\n```"
        )
    }
]

prompt = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True
)

inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

with torch.no_grad():
    outputs = model.generate(
        **inputs,
        max_new_tokens=256,
        do_sample=False
    )

new_tokens = outputs[0][inputs["input_ids"].shape[1]:]
print(tokenizer.decode(new_tokens, skip_special_tokens=True))

The example demonstrates the intended interface. The actual prediction is not guaranteed to match any illustrative output.

Training Details

Training Data

The training dataset is derived from OWASP Benchmark Java.

• Dataset repository: dilekmandal/owasp-benchmark-java-security
• Total examples: 21,041
• Training examples: 16,849
• Validation examples: 2,096
• Test examples: 2,096

The examples use a chat-style structure with Java source code as input and a structured classification target.

The dataset includes vulnerable and non-vulnerable samples across multiple CWE categories. Refer to the dataset card and the original OWASP Benchmark project for further context.

Preprocessing

The source data was converted to JSON Lines (JSONL) examples using chat messages. The instruction requests JSON output containing vulnerable, cwe, and category.

Dataset auditing was performed before model publication. The identified cross-split code overlaps are documented in the limitations section above.

Training Procedure

The model was fine-tuned using LoRA. Detailed hyperparameters, training duration, and compute measurements are not reported here because they have not been independently documented in this card.

Evaluation

Testing Data

The initial evaluation used the test split of the prepared dataset. During dataset auditing, exact normalized-code overlaps and shared code families were identified across the training, validation, and test splits.

Consequently, the initial test results may overestimate the model's generalization performance.

Evaluation Metrics

The following metrics are relevant for evaluating this classification task:

Accuracy

Precision

Recall

F1 score

CWE-level evaluation is also necessary to assess whether the model correctly identifies specific vulnerability types, rather than only predicting whether a code sample is vulnerable.

Evaluation Status

Initial local evaluation experiments have been conducted. However, due to cross-split code overlap and shared code families, the existing results are not reported here as reliable estimates of generalization performance.

Further evaluation is needed using leakage-resistant, code-family-based data splits and an independent test set. CWE-level error analysis is also planned to better understand the model's strengths and limitations.

The model should be considered an experimental research artifact, not a validated security analysis tool.

Technical Specifications

Model Architecture and Objective

The adapter fine-tunes a causal language model using parameter-efficient LoRA adaptation. The objective is to generate structured vulnerability-classification responses from Java source code.

Compute Infrastructure

Hardware, training duration, energy consumption, and software versions used during the complete training run have not been fully documented in this model card.

Licensing

Review the base model's license and the original dataset's licensing terms before redistributing or using this adapter. No license is asserted here until compatibility and redistribution permissions have been verified.

• Base model: Qwen2.5-Coder-7B-Instruct
• Original benchmark: OWASP Benchmark Java

Acknowledgments

This project uses the OWASP Benchmark Java project as a source of benchmark data and Qwen2.5-Coder-7B-Instruct as its base model.

Model Card Authors

Dilek Mandal