samliumay/gemma_based_mini_turkish_cyber_security_controls_assistant

Gemma-Based Mini Turkish Cyber Security Controls Assistant

This is a personal small-model experiment fine-tuned from Gemma 3 1B IT with Unsloth. It is designed to answer Turkish questions about cyber security controls, risk-based control selection, NIST SP 800-53, Zero Trust, cloud security, IAM, logging, incident response, and related defensive security topics.

Model ID: samliumay/gemma_based_mini_turkish_cyber_security_controls_assistant

Training data

The model was fine-tuned on samliumay/turkish_cyber_security_controls_dataset, which contains 800 synthetic Turkish user-assistant conversation pairs (1,600 messages). The dataset is intended for defensive security education and instruction tuning and contains no real user conversations or personal data.

Fine-tuning was performed with Unsloth in Google Colab on an NVIDIA Tesla T4.

Evaluation

Turkish MMLU-style benchmark

The LoRA adapter was evaluated after loading it over the exact gemma3:1b-it-fp16 base model in Ollama. The evaluation used all 6,200 multiple-choice questions across 62 sections from alibayram/yapay_zeka_turkce_mmlu_model_cevaplari. The same run was repeated on the base model without the adapter.

Model	Correct	Accuracy	95% Wilson CI	Invalid responses
This LoRA + Gemma 3 1B IT FP16	1,843 / 6,200	29.73%	28.60%–30.88%	67
Gemma 3 1B IT FP16 base	1,909 / 6,200	30.79%	29.65%–31.95%	11

In the paired comparison, the adapter scored 1.06 percentage points lower than the base model (66 fewer correct answers). The paired 95% interval for the difference was -2.01 to -0.12 percentage points, with an exact two-sided McNemar p-value of 0.0298. The adapter was solely correct on 415 questions, while the base model was solely correct on 481.

Evaluation details:

• Date: 2026-07-23
• Evaluator: a local, reproducible adaptation of olcum.py
• Runtime: Ollama 0.32.1 on Windows 11, 100% GPU execution
• Model precision: GGUF F16, approximately 999.89M parameters
• Generation: seed 42, at most 42 generated tokens, thinking disabled, temperature 1, top-k 64, top-p 0.95
• Scoring: strict option-letter/explicit-answer parsing; ambiguous output was counted as incorrect
• Dataset SHA-256: da37123640644a5ce1452a39e96261238977ad9f5d070f1d38c6879aecaa1e95

This strict score is not directly interchangeable with the public leaderboard score. The original evaluator uses a sentence-embedding fallback for responses that are not formatted as an option letter; this evaluation deliberately did not infer an answer from ambiguous prose. The adapter produced more such responses than the base model (67 versus 11), so the comparison measures both general knowledge and compliance with the requested answer format.

Most importantly, this dataset is a broad Turkish educational benchmark and does not contain a dedicated cybersecurity section. These results measure general Turkish multiple-choice performance, not the quality of the model's cybersecurity-control guidance. The complete scripts, raw responses, section-level results, and paired comparison are available in the local benchmark_of_mini_model_for_cyber_security folder.

Run locally on Linux with an NVIDIA GPU

Make sure nvidia-smi works, then create a Python environment and install the dependencies:

python3 -m venv .venv
source .venv/bin/activate
python -m pip install -U pip
pip install -U torch "transformers>=4.50" accelerate safetensors

Run the merged Transformers model:

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "samliumay/gemma_based_mini_turkish_cyber_security_controls_assistant"

if not torch.cuda.is_available():
    raise RuntimeError("An NVIDIA GPU with a working CUDA-enabled PyTorch build is required.")

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.float16,
    device_map="auto",
).eval()

messages = [
    {"role": "user", "content": "Zero Trust mimarisi nedir?"},
]
inputs = tokenizer.apply_chat_template(
    messages,
    add_generation_prompt=True,
    tokenize=True,
    return_dict=True,
    return_tensors="pt",
).to(model.device)

with torch.inference_mode():
    output_ids = model.generate(
        **inputs,
        max_new_tokens=256,
        do_sample=True,
        temperature=0.7,
        top_p=0.9,
    )

new_tokens = output_ids[0, inputs["input_ids"].shape[-1]:]
print(tokenizer.decode(new_tokens, skip_special_tokens=True))

Limitations

This experimental model can hallucinate, return outdated information, or misinterpret standards and control identifiers. Its responses are general educational guidance, not a substitute for current authoritative standards, technical testing, legal advice, or review by a qualified security professional. Do not use it as the sole basis for production security or compliance decisions.

License

This model is a derivative of Gemma and is subject to the Gemma Terms of Use and Gemma Prohibited Use Policy. The training dataset is released under the MIT License.