Training complete: Granite 20B SecureCode (3 epochs, loss 1.639)

README.md

@@ -1,732 +1,60 @@
-# IBM Granite 20B Code - SecureCode Edition
-
-<div align="center">
-
-[![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://opensource.org/licenses/Apache-2.0)
-[![Training Dataset](https://img.shields.io/badge/dataset-SecureCode%20v2.0-green.svg)](https://huggingface.co/datasets/scthornton/securecode-v2)
-[![Base Model](https://img.shields.io/badge/base-Granite%2020B%20Code-orange.svg)](https://huggingface.co/ibm-granite/granite-20b-code-instruct-8k)
-[![perfecXion.ai](https://img.shields.io/badge/by-perfecXion.ai-purple.svg)](https://perfecxion.ai)
-
-**🏢 Enterprise-scale security intelligence with IBM trust**
-
-The most powerful model in the SecureCode collection. When you need maximum code understanding, complex reasoning, and IBM's enterprise-grade reliability.
-
-[🤗 Model Hub](https://huggingface.co/scthornton/granite-20b-code-securecode) | [📊 Dataset](https://huggingface.co/datasets/scthornton/securecode-v2) | [💻 perfecXion.ai](https://perfecxion.ai) | [📚 Collection](https://huggingface.co/collections/scthornton/securecode)
-
-</div>
-
----
-
-## 🎯 Quick Decision Guide
-
-**Choose This Model If:**
-- ✅ You need **maximum code understanding** and security reasoning capability
-- ✅ You're analyzing **complex enterprise architectures** with intricate attack surfaces
-- ✅ You require **IBM enterprise trust** and brand recognition
-- ✅ You have **datacenter infrastructure** (48GB+ GPU)
-- ✅ You're conducting **professional security audits** requiring comprehensive analysis
-- ✅ You need the **most sophisticated** security intelligence in the collection
-
-**Consider Smaller Models If:**
-- ⚠️ You're on consumer hardware (→ Llama 3B, Qwen 7B)
-- ⚠️ You prioritize inference speed over depth (→ Qwen 7B/14B)
-- ⚠️ You're building IDE tools needing fast response (→ Llama 3B, DeepSeek 6.7B)
-- ⚠️ Budget is primary concern (→ any 7B/13B model)
-
----
-
-## 📊 Collection Positioning
-
-| Model | Size | Best For | Hardware | Inference Speed | Unique Strength |
-|-------|------|----------|----------|-----------------|-----------------|
-| Llama 3.2 3B | 3B | Consumer deployment | 8GB RAM | ⚡⚡⚡ Fastest | Most accessible |
-| DeepSeek 6.7B | 6.7B | Security-optimized baseline | 16GB RAM | ⚡⚡ Fast | Security architecture |
-| Qwen 7B | 7B | Best code understanding | 16GB RAM | ⚡⚡ Fast | Best-in-class 7B |
-| CodeGemma 7B | 7B | Google ecosystem | 16GB RAM | ⚡⚡ Fast | Instruction following |
-| CodeLlama 13B | 13B | Enterprise trust | 24GB RAM | ⚡ Medium | Meta brand, proven |
-| Qwen 14B | 14B | Advanced analysis | 32GB RAM | ⚡ Medium | 128K context window |
-| StarCoder2 15B | 15B | Multi-language specialist | 32GB RAM | ⚡ Medium | 600+ languages |
-| **Granite 20B** | **20B** | **Enterprise-scale** | **48GB RAM** | **Medium** | **IBM trust, largest, most capable** |
-
-**This Model's Position:** The flagship. Maximum security intelligence, enterprise-grade reliability, IBM brand trust. For when quality matters more than speed.
-
----
-
-## 🚨 The Problem This Solves
-
-**Critical enterprise security gaps require sophisticated analysis.** When a breach costs **$4.45 million on average** (IBM 2024 Cost of Data Breach Report) and 45% of AI-generated code contains vulnerabilities, enterprises need the most capable security analysis available.
-
-**Real-world enterprise impact:**
-- **Equifax** (SQL injection): $425 million settlement + 13-year brand recovery
-- **Capital One** (SSRF): 100 million customer records, $80M fine, 2 years of remediation
-- **SolarWinds** (supply chain): 18,000 organizations compromised, $18M settlement
-- **LastPass** (cryptographic failures): 30M users affected, company reputation destroyed
-
-**IBM Granite 20B SecureCode Edition** provides the deepest security analysis available in the open-source ecosystem, backed by IBM's enterprise heritage and trust.
-
----
-
-## 💡 What is This?
-
-This is **IBM Granite 20B Code Instruct** fine-tuned on the **SecureCode v2.0 dataset** - IBM's enterprise-grade code model enhanced with production-grade security expertise covering the complete OWASP Top 10:2025.
-
-IBM Granite models are built on IBM's 40+ years of enterprise software experience, trained on **3.5+ trillion tokens** of code and technical data, with a focus on enterprise deployment reliability.
-
-Combined with SecureCode training, this model delivers:
-
-✅ **Maximum security intelligence** - 20B parameters for deep, nuanced analysis
-✅ **Enterprise-grade reliability** - IBM's proven track record and support ecosystem
-✅ **Comprehensive vulnerability detection** across complex architectures
-✅ **Production-ready trust** - Permissive Apache 2.0 license
-✅ **Advanced reasoning** - Handles multi-layered attack chain analysis
-
-**The Result:** The most capable security-aware code model in the open-source ecosystem.
-
-**Why IBM Granite 20B?** This model is the enterprise choice:
-- 🏢 **IBM enterprise heritage** - 40+ years of enterprise software leadership
-- 🔐 **Largest in collection** - 20B parameters = maximum reasoning capability
-- 📋 **Enterprise compliance ready** - Designed for regulated industries
-- ⚖️ **Apache 2.0 licensed** - Full commercial freedom
-- 🎯 **Security-first training** - Built for mission-critical applications
-- 🌍 **Broad language support** - 116+ programming languages
-
-Perfect for Fortune 500 companies, financial services, healthcare, government, and any organization where security analysis quality is paramount.
-
----
-
-## 🔐 Security Training Coverage
-
-### Real-World Vulnerability Distribution
-
-Trained on 1,209 security examples with real CVE grounding:
-
-| OWASP Category | Examples | Real Incidents |
-|----------------|----------|----------------|
-| **Broken Access Control** | 224 | Equifax, Facebook, Uber |
-| **Authentication Failures** | 199 | SolarWinds, Okta, LastPass |
-| **Injection Attacks** | 125 | Capital One, Yahoo, LinkedIn |
-| **Cryptographic Failures** | 115 | LastPass, Adobe, Dropbox |
-| **Security Misconfiguration** | 98 | Tesla, MongoDB, Elasticsearch |
-| **Vulnerable Components** | 87 | Log4Shell, Heartbleed, Struts |
-| **Identification/Auth Failures** | 84 | Twitter, GitHub, Reddit |
-| **Software/Data Integrity** | 78 | SolarWinds, Codecov, npm |
-| **Logging Failures** | 71 | Various incident responses |
-| **SSRF** | 69 | Capital One, Shopify |
-| **Insecure Design** | 59 | Architectural flaws |
-
-### Enterprise-Grade Multi-Language Support
-
-Fine-tuned on security examples across:
-- **Python** (Django, Flask, FastAPI) - 280 examples
-- **JavaScript/TypeScript** (Express, NestJS, React) - 245 examples
-- **Java** (Spring Boot, Jakarta EE) - 178 examples
-- **Go** (Gin, Echo, standard library) - 145 examples
-- **PHP** (Laravel, Symfony) - 112 examples
-- **C#** (ASP.NET Core, .NET 6+) - 89 examples
-- **Ruby** (Rails, Sinatra) - 67 examples
-- **Rust** (Actix, Rocket, Axum) - 45 examples
-- **C/C++** (Memory safety patterns) - 28 examples
-- **Plus 107+ additional languages from Granite's base training**
-
----
-
-## 🎯 Deployment Scenarios
-
-### Scenario 1: Enterprise Security Audit Platform
-
-**Professional security assessments for Fortune 500 clients.**
-
-**Hardware:** Datacenter GPU (A100 80GB or 2x A100 40GB)
-**Throughput:** 10-15 comprehensive audits/day
-**Use Case:** Professional security consulting
-
-**Value Proposition:**
-- Identify vulnerabilities human auditors miss
-- Consistent, comprehensive OWASP coverage
-- Scales expert security knowledge
-- Reduces audit time by 60-70%
-
-**ROI:** A single prevented breach pays for years of infrastructure. Typical large enterprise security audit costs $150K-500K. This model can handle preliminary analysis, allowing human experts to focus on novel vulnerabilities and strategic recommendations.
-
----
-
-### Scenario 2: Financial Services Security Platform
-
-**Regulatory compliance and security for banking applications.**
-
-**Hardware:** Private cloud A100 cluster
-**Compliance:** SOC 2, PCI-DSS, GDPR, CCPA
-**Use Case:** Pre-deployment security validation
-
-**Regulatory Benefits:**
-- Automated OWASP Top 10 verification
-- Audit trail generation
-- Compliance report automation
-- Reduces regulatory risk
-
-**ROI:** Regulatory fines cost millions. **Capital One:** $80M fine. **Equifax:** $425M settlement. Preventing one major breach justifies entire deployment.
-
----
-
-### Scenario 3: Healthcare Application Security
-
-**HIPAA-compliant code review for medical systems.**
-
-**Hardware:** Secure private deployment
-**Compliance:** HIPAA, HITECH, FDA software validation
-**Use Case:** Medical device and EHR security
-
-**Critical Healthcare Requirements:**
-- Patient data protection (HIPAA)
-- Audit logging and compliance
-- Cryptographic requirements
-- Access control verification
-
-**Impact:** Healthcare breaches average **$10.93 million per incident** (IBM 2024). Single prevented breach pays for multi-year deployment.
-
----
-
-### Scenario 4: Government & Defense Applications
-
-**Security analysis for critical infrastructure.**
-
-**Hardware:** Air-gapped secure environment
-**Clearance:** Can be deployed in classified environments
-**Use Case:** Critical infrastructure security
-
-**Government Benefits:**
-- No external dependencies (fully local)
-- Apache 2.0 license (government-friendly)
-- IBM enterprise support available
-- Meets government security standards
-
----
-
-## 📊 Training Details
-
-| Parameter | Value | Why This Matters |
-|-----------|-------|------------------|
-| **Base Model** | ibm-granite/granite-20b-code-instruct-8k | IBM's enterprise-grade foundation |
-| **Fine-tuning Method** | LoRA (Low-Rank Adaptation) | Efficient training, preserves base capabilities |
-| **Training Dataset** | [SecureCode v2.0](https://huggingface.co/datasets/scthornton/securecode-v2) | 100% incident-grounded, expert-validated |
-| **Dataset Size** | 841 training examples | Focused on quality over quantity |
-| **Training Epochs** | 3 | Optimal convergence without overfitting |
-| **LoRA Rank (r)** | 16 | Balanced parameter efficiency |
-| **LoRA Alpha** | 32 | Learning rate scaling factor |
-| **Learning Rate** | 2e-4 | Standard for LoRA fine-tuning |
-| **Quantization** | 4-bit (bitsandbytes) | Enables efficient training |
-| **Trainable Parameters** | ~105M (0.525% of 20B total) | Minimal parameters, maximum impact |
-| **Total Parameters** | 20B | Maximum reasoning capability |
-| **Context Window** | 8K tokens | Enterprise file analysis |
-| **GPU Used** | NVIDIA A100 40GB | Enterprise training infrastructure |
-| **Training Time** | ~12-14 hours (estimated) | Deep security learning |
-
-### Training Methodology
-
-**LoRA (Low-Rank Adaptation)** was chosen for enterprise reliability:
-1. **Efficiency:** Trains only 0.525% of model parameters (105M vs 20B)
-2. **Quality:** Preserves IBM Granite's enterprise capabilities
-3. **Deployability:** Can be deployed alongside base model for versioning
-
-**4-bit Quantization** enables efficient training while maintaining enterprise-grade quality.
-
-**IBM Granite Foundation:** Built on IBM's 40+ years of enterprise software experience, optimized for:
-- Reliability and consistency
-- Enterprise deployment patterns
-- Regulatory compliance requirements
-- Long-term support and stability
-
----
-
-## 🚀 Usage
-
-### Quick Start
-
-```python
-from transformers import AutoModelForCausalLM, AutoTokenizer
-from peft import PeftModel
-
-# Load IBM Granite base model
-base_model = "ibm-granite/granite-20b-code-instruct-8k"
-model = AutoModelForCausalLM.from_pretrained(
-    base_model,
-    device_map="auto",
-    torch_dtype="auto",
-    trust_remote_code=True
-)
-tokenizer = AutoTokenizer.from_pretrained(base_model, trust_remote_code=True)
-
-# Load SecureCode LoRA adapter
-model = PeftModel.from_pretrained(model, "scthornton/granite-20b-code-securecode")
-
-# Enterprise security analysis
-prompt = """### User:
-Conduct a comprehensive security audit of this enterprise authentication system. Analyze for:
-1. OWASP Top 10 vulnerabilities
-2. Attack chain opportunities
-3. Compliance gaps (SOC 2, PCI-DSS)
-4. Architectural weaknesses
-
-```python
-# Enterprise SSO Implementation
-class EnterpriseAuthService:
-    def __init__(self):
-        self.secret = os.getenv('JWT_SECRET')
-        self.db = DatabasePool()
-
-    async def authenticate(self, credentials):
-        user = await self.db.query(
-            f"SELECT * FROM users WHERE email='{credentials.email}' AND password='{credentials.password}'"
-        )
-        if user:
-            token = jwt.encode({'user_id': user.id}, self.secret)
-            return {'token': token, 'success': True}
-        return {'success': False}
-
-    async def verify_token(self, token):
-        try:
-            payload = jwt.decode(token, self.secret, algorithms=['HS256'])
-            return payload
-        except:
-            return None
-```
-
-### Assistant:
-"""
-
-inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
-outputs = model.generate(
-    **inputs,
-    max_new_tokens=4096,
-    temperature=0.2,  # Lower temperature for precise enterprise analysis
-    top_p=0.95,
-    do_sample=True
-)
-
-response = tokenizer.decode(outputs[0], skip_special_tokens=True)
-print(response)
-```
-
----
-
-### Enterprise Deployment (4-bit Quantization)
-
-```python
-from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
-from peft import PeftModel
-
-# 4-bit quantization - runs on 40GB GPU
-bnb_config = BitsAndBytesConfig(
-    load_in_4bit=True,
-    bnb_4bit_use_double_quant=True,
-    bnb_4bit_quant_type="nf4",
-    bnb_4bit_compute_dtype="bfloat16"
-)
-
-model = AutoModelForCausalLM.from_pretrained(
-    "ibm-granite/granite-20b-code-instruct-8k",
-    quantization_config=bnb_config,
-    device_map="auto",
-    trust_remote_code=True
-)
-
-model = PeftModel.from_pretrained(model, "scthornton/granite-20b-code-securecode")
-tokenizer = AutoTokenizer.from_pretrained("ibm-granite/granite-20b-code-instruct-8k", trust_remote_code=True)
-
-# Enterprise-ready: Runs on A100 40GB, A100 80GB, or 2x RTX 4090
-```
-
----
-
-### Multi-GPU Deployment (Maximum Performance)
-
-```python
-from transformers import AutoModelForCausalLM, AutoTokenizer
-from peft import PeftModel
-import torch
-
-# Load across multiple GPUs for maximum throughput
-model = AutoModelForCausalLM.from_pretrained(
-    "ibm-granite/granite-20b-code-instruct-8k",
-    device_map="balanced",  # Distribute across available GPUs
-    torch_dtype=torch.bfloat16,
-    trust_remote_code=True
-)
-
-model = PeftModel.from_pretrained(model, "scthornton/granite-20b-code-securecode")
-tokenizer = AutoTokenizer.from_pretrained("ibm-granite/granite-20b-code-instruct-8k", trust_remote_code=True)
-
-# Optimal for: 2x A100, 4x RTX 4090, or enterprise GPU clusters
-# Throughput: 2-3x faster than single GPU
-```
-
----
-
-## 📈 Performance & Benchmarks
-
-### Hardware Requirements
-
-| Deployment | RAM | GPU VRAM | Tokens/Second | Latency (4K response) | Cost/Month |
-|-----------|-----|----------|---------------|----------------------|------------|
-| **4-bit Quantized** | 40GB | 32GB | ~35 tok/s | ~115 seconds | $0 (on-prem) or $800-1200 (cloud) |
-| **8-bit Quantized** | 64GB | 48GB | ~45 tok/s | ~90 seconds | $0 (on-prem) or $1200-1800 (cloud) |
-| **Full Precision (bf16)** | 96GB | 80GB | ~60 tok/s | ~67 seconds | $0 (on-prem) or $2000-3000 (cloud) |
-| **Multi-GPU (2x A100)** | 128GB | 160GB | ~120 tok/s | ~33 seconds | Enterprise only |
-
-### Real-World Performance
-
-**Tested on A100 40GB** (enterprise GPU):
-- **Tokens/second:** ~35 tok/s (4-bit), ~55 tok/s (full precision)
-- **Cold start:** ~8 seconds
-- **Memory usage:** 28GB (4-bit), 42GB (full precision)
-- **Throughput:** 200-300 comprehensive analyses per day
-
-**Tested on 2x A100 80GB** (multi-GPU):
-- **Tokens/second:** ~110-120 tok/s
-- **Cold start:** ~6 seconds
-- **Throughput:** 500+ analyses per day
-
-### Security Analysis Quality
-
-**The differentiator:** Granite 20B provides the deepest, most nuanced security analysis:
-- Identifies **15-25% more vulnerabilities** than 7B models in complex code
-- Detects **multi-step attack chains** that smaller models miss
-- Provides **enterprise-grade operational guidance** with compliance mapping
-- **Reduces false positives** through sophisticated reasoning
-
 ---
-
-## 💰 Cost Analysis
-
-### Total Cost of Ownership (TCO) - 1 Year
-
-**Option 1: On-Premise (Dedicated Server)**
-- Hardware: 2x A100 40GB - $20,000 (one-time capital expense)
-- Server infrastructure: $5,000
-- Electricity: ~$2,400/year
-- **Total Year 1:** $27,400
-- **Total Year 2+:** $2,400/year
-
-**Option 2: Cloud GPU (AWS/GCP/Azure)**
-- Instance: A100 40GB (p4d.xlarge)
-- Cost: ~$3.50/hour
-- Usage: 160 hours/month (enterprise team)
-- **Total Year 1:** $6,720/year
-
-**Option 3: Enterprise GPT-4 (for comparison)**
-- Cost: $30/1M input tokens, $60/1M output tokens
-- Usage: 500M input + 500M output tokens/year
-- **Total Year 1:** $45,000/year
-
-**Option 4: Professional Security Audits (for comparison)**
-- Average enterprise security audit: $150,000-500,000
-- Frequency: Quarterly (4x/year)
-- **Total Year 1:** $600,000-2,000,000
-
-**ROI Winner:** On-premise deployment pays for itself with **1-2 prevented security audits** or **preventing a single breach** (average cost: $4.45M).
-
+library_name: peft
+license: apache-2.0
+base_model: ibm-granite/granite-20b-code-instruct-8k
+tags:
+- base_model:adapter:ibm-granite/granite-20b-code-instruct-8k
+- lora
+- transformers
+pipeline_tag: text-generation
+model-index:
+- name: granite-20b-code-securecode
+  results: []
 ---
 
-## 🎯 Use Cases & Examples
+<!-- This model card has been generated automatically according to the information the Trainer had access to. You
+should probably proofread and complete it, then remove this comment. -->
 
-### 1. Enterprise Security Architecture Review
+# granite-20b-code-securecode
 
-Analyze complex microservices platforms:
+This model is a fine-tuned version of [ibm-granite/granite-20b-code-instruct-8k](https://huggingface.co/ibm-granite/granite-20b-code-instruct-8k) on the None dataset.
 
-```python
-prompt = """### User:
-Conduct a comprehensive security architecture review of this fintech payment platform. Analyze:
-1. Service-to-service authentication security
-2. Data flow security boundaries
-3. Compliance with PCI-DSS requirements
-4. Attack surface analysis
-5. Defense-in-depth gaps
+## Model description
 
-[Include microservices code across auth-service, payment-service, notification-service]
+More information needed
 
-### Assistant:
-"""
-```
-
-**Model Response:** Provides 20-30 page comprehensive analysis with specific vulnerability findings, attack chain scenarios, compliance gaps, and remediation priorities.
-
----
+## Intended uses & limitations
 
-### 2. Regulatory Compliance Validation
+More information needed
 
-Validate code against regulatory requirements:
+## Training and evaluation data
 
-```python
-prompt = """### User:
-Analyze this healthcare EHR system for HIPAA compliance. Verify:
-1. Patient data encryption (at rest and in transit)
-2. Access control and audit logging
-3. Data retention policies
-4. Breach notification capabilities
-5. Business Associate Agreement requirements
+More information needed
 
-[Include EHR codebase]
+## Training procedure
 
-### Assistant:
-"""
-```
+### Training hyperparameters
 
-**Model Response:** Detailed compliance mapping, gap analysis, and remediation roadmap.
-
----
-
-### 3. Supply Chain Security Analysis
-
-Analyze third-party dependencies and integrations:
-
-```python
-prompt = """### User:
-Perform a supply chain security analysis of this application:
-1. Third-party library vulnerabilities
-2. Dependency confusion risks
-3. Code injection via dependencies
-4. Malicious package detection
-5. License compliance issues
-
-[Include package.json, requirements.txt, go.mod]
-
-### Assistant:
-"""
-```
-
-**Model Response:** Comprehensive supply chain risk assessment with mitigation strategies.
-
----
-
-### 4. Advanced Penetration Testing Guidance
-
-Develop sophisticated attack scenarios:
-
-```python
-prompt = """### User:
-Design a comprehensive penetration testing strategy for this enterprise web application. Include:
-1. Attack surface enumeration
-2. Vulnerability prioritization
-3. Multi-stage attack chains
-4. Privilege escalation paths
-5. Data exfiltration scenarios
-6. Post-exploitation persistence
-
-### Assistant:
-"""
-```
-
-**Model Response:** Professional pentesting methodology with specific attack vectors and validation procedures.
-
----
-
-## ⚠️ Limitations & Transparency
-
-### What This Model Does Well
-✅ Maximum code understanding and security reasoning
-✅ Complex attack chain analysis and enterprise architecture review
-✅ Detailed operational guidance and compliance mapping
-✅ Sophisticated multi-layered vulnerability detection
-✅ Enterprise-scale codebase analysis
-✅ IBM enterprise trust and reliability
-
-### What This Model Doesn't Do
-❌ **Not a security scanner** - Use tools like Semgrep, CodeQL, Snyk, or Veracode
-❌ **Not a penetration testing tool** - Cannot perform active exploitation or network scanning
-❌ **Not legal/compliance advice** - Consult security and legal professionals
-❌ **Not a replacement for security experts** - Critical systems need professional security review and audits
-❌ **Not real-time threat intelligence** - Training data frozen at Dec 2024
-
-### Known Issues & Constraints
-- **Inference latency:** Larger model means slower responses (35-60 tok/s vs 100+ tok/s for smaller models)
-- **Hardware requirements:** Requires enterprise GPU infrastructure (40GB+ VRAM)
-- **Detailed analysis:** May generate very comprehensive responses (3000-4000 tokens)
-- **Cost consideration:** Higher deployment cost than smaller models
-- **Context window:** 8K tokens (vs 128K for Qwen models)
-
-### Appropriate Use
-✅ Enterprise security audits and professional assessments
-✅ Regulatory compliance validation
-✅ Critical infrastructure security review
-✅ Financial services and healthcare applications
-✅ Government and defense security analysis
-
-### Inappropriate Use
-❌ Sole validation for production deployments (use comprehensive testing)
-❌ Replacement for professional security audits
-❌ Active exploitation or penetration testing without authorization
-❌ Consumer applications (too large, use smaller models)
-
----
+The following hyperparameters were used during training:
+- learning_rate: 0.0002
+- train_batch_size: 1
+- eval_batch_size: 8
+- seed: 42
+- gradient_accumulation_steps: 16
+- total_train_batch_size: 16
+- optimizer: Use paged_adamw_8bit with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
+- lr_scheduler_type: cosine
+- lr_scheduler_warmup_steps: 100
+- num_epochs: 3
 
-## 🔬 Dataset Information
+### Training results
 
-This model was trained on **[SecureCode v2.0](https://huggingface.co/datasets/scthornton/securecode-v2)**, a production-grade security dataset with:
 
-- **1,209 total examples** (841 train / 175 validation / 193 test)
-- **100% incident grounding** - every example tied to real CVEs or security breaches
-- **11 vulnerability categories** - complete OWASP Top 10:2025 coverage
-- **11 programming languages** - from Python to Rust
-- **4-turn conversational structure** - mirrors real developer-AI workflows
-- **100% expert validation** - reviewed by independent security professionals
-
-See the [full dataset card](https://huggingface.co/datasets/scthornton/securecode-v2) and [research paper](https://perfecxion.ai/articles/securecode-v2-dataset-paper.html) for complete details.
-
----
-
-## 🏢 About perfecXion.ai
-
-[perfecXion.ai](https://perfecxion.ai) is dedicated to advancing AI security through research, datasets, and production-grade security tooling.
-
-**Connect:**
-- Website: [perfecxion.ai](https://perfecxion.ai)
-- Research: [perfecxion.ai/research](https://perfecxion.ai/research)
-- Knowledge Hub: [perfecxion.ai/knowledge](https://perfecxion.ai/knowledge)
-- GitHub: [@scthornton](https://github.com/scthornton)
-- HuggingFace: [@scthornton](https://huggingface.co/scthornton)
-- Email: scott@perfecxion.ai
-
----
-
-## 📄 License
-
-**Model License:** Apache 2.0 (permissive - use in commercial applications)
-**Dataset License:** CC BY-NC-SA 4.0 (non-commercial with attribution)
-
-### What You CAN Do
-✅ Use this model commercially in production applications
-✅ Fine-tune further for your specific use case
-✅ Deploy in enterprise environments
-✅ Integrate into commercial products
-✅ Distribute and modify the model weights
-✅ Charge for services built on this model
-✅ Use in government and regulated industries
-
-### What You CANNOT Do with the Dataset
-❌ Sell or redistribute the raw SecureCode v2.0 dataset commercially
-❌ Use the dataset to train commercial models without releasing under the same license
-❌ Remove attribution or claim ownership of the dataset
-
-For commercial dataset licensing or custom training, contact: scott@perfecxion.ai
-
----
-
-## 📚 Citation
-
-If you use this model in your research or applications, please cite:
-
-```bibtex
-@misc{thornton2025securecode-granite20b,
-  title={IBM Granite 20B Code - SecureCode Edition},
-  author={Thornton, Scott},
-  year={2025},
-  publisher={perfecXion.ai},
-  url={https://huggingface.co/scthornton/granite-20b-code-securecode},
-  note={Fine-tuned on SecureCode v2.0: https://huggingface.co/datasets/scthornton/securecode-v2}
-}
-
-@misc{thornton2025securecode-dataset,
-  title={SecureCode v2.0: A Production-Grade Dataset for Training Security-Aware Code Generation Models},
-  author={Thornton, Scott},
-  year={2025},
-  month={January},
-  publisher={perfecXion.ai},
-  url={https://perfecxion.ai/articles/securecode-v2-dataset-paper.html},
-  note={Dataset: https://huggingface.co/datasets/scthornton/securecode-v2}
-}
-```
-
----
-
-## 🙏 Acknowledgments
-
-- **IBM Research** for the exceptional Granite code models and enterprise commitment
-- **OWASP Foundation** for maintaining the Top 10 vulnerability taxonomy
-- **MITRE Corporation** for the CVE database and vulnerability research
-- **Security research community** for responsible disclosure practices
-- **Hugging Face** for model hosting and inference infrastructure
-- **Enterprise security teams** who validated this model in production environments
-
----
-
-## 🤝 Contributing
-
-Found a security issue or have suggestions for improvement?
-
-- 🐛 **Report issues:** [GitHub Issues](https://github.com/scthornton/securecode-models/issues)
-- 💬 **Discuss improvements:** [HuggingFace Discussions](https://huggingface.co/scthornton/granite-20b-code-securecode/discussions)
-- 📧 **Contact:** scott@perfecxion.ai
-
-### Community Contributions Welcome
-
-Especially interested in:
-- **Enterprise deployment case studies**
-- **Benchmark evaluations** on industry security datasets
-- **Compliance validation** (PCI-DSS, HIPAA, SOC 2)
-- **Performance optimization** for specific enterprise hardware
-- **Integration examples** with enterprise security platforms
-
----
-
-## 🔗 SecureCode Model Collection
-
-Explore other SecureCode fine-tuned models optimized for different use cases:
-
-### Entry-Level Models (3-7B)
-- **[llama-3.2-3b-securecode](https://huggingface.co/scthornton/llama-3.2-3b-securecode)**
-  - **Best for:** Consumer hardware, IDE integration, education
-  - **Hardware:** 8GB RAM minimum
-  - **Unique strength:** Most accessible
-
-- **[deepseek-coder-6.7b-securecode](https://huggingface.co/scthornton/deepseek-coder-6.7b-securecode)**
-  - **Best for:** Security-optimized baseline
-  - **Hardware:** 16GB RAM
-  - **Unique strength:** Security-first architecture
-
-- **[qwen2.5-coder-7b-securecode](https://huggingface.co/scthornton/qwen2.5-coder-7b-securecode)**
-  - **Best for:** Best code understanding in 7B class
-  - **Hardware:** 16GB RAM
-  - **Unique strength:** 128K context, best-in-class
-
-- **[codegemma-7b-securecode](https://huggingface.co/scthornton/codegemma-7b-securecode)**
-  - **Best for:** Google ecosystem, instruction following
-  - **Hardware:** 16GB RAM
-  - **Unique strength:** Google brand, strong completion
-
-### Mid-Range Models (13-15B)
-- **[codellama-13b-securecode](https://huggingface.co/scthornton/codellama-13b-securecode)**
-  - **Best for:** Enterprise trust, Meta brand
-  - **Hardware:** 24GB RAM
-  - **Unique strength:** Proven track record
-
-- **[qwen2.5-coder-14b-securecode](https://huggingface.co/scthornton/qwen2.5-coder-14b-securecode)**
-  - **Best for:** Advanced code analysis
-  - **Hardware:** 32GB RAM
-  - **Unique strength:** 128K context window
-
-- **[starcoder2-15b-securecode](https://huggingface.co/scthornton/starcoder2-15b-securecode)**
-  - **Best for:** Multi-language projects (600+ languages)
-  - **Hardware:** 32GB RAM
-  - **Unique strength:** Broadest language support
-
-### Enterprise-Scale Models (20B+)
-- **[granite-20b-code-securecode](https://huggingface.co/scthornton/granite-20b-code-securecode)** ⭐ (YOU ARE HERE)
-  - **Best for:** Enterprise-scale, IBM trust, maximum capability
-  - **Hardware:** 48GB RAM
-  - **Unique strength:** Largest model, deepest analysis
-
-**View Complete Collection:** [SecureCode Models](https://huggingface.co/collections/scthornton/securecode)
-
----
-
-<div align="center">
-
-**Built with ❤️ for secure enterprise software**
-
-[perfecXion.ai](https://perfecxion.ai) | [Research](https://perfecxion.ai/research) | [Knowledge Hub](https://perfecxion.ai/knowledge) | [Contact](mailto:scott@perfecxion.ai)
-
----
 
-*Maximum security intelligence. Enterprise trust. IBM heritage.*
+### Framework versions
 
-</div>
+- PEFT 0.18.1
+- Transformers 4.57.6
+- Pytorch 2.7.1+cu128
+- Datasets 4.5.0
+- Tokenizers 0.22.2

adapter_config.json

@@ -0,0 +1,43 @@
+{
+  "alora_invocation_tokens": null,
+  "alpha_pattern": {},
+  "arrow_config": null,
+  "auto_mapping": null,
+  "base_model_name_or_path": "ibm-granite/granite-20b-code-instruct-8k",
+  "bias": "none",
+  "corda_config": null,
+  "ensure_weight_tying": false,
+  "eva_config": null,
+  "exclude_modules": null,
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 16,
+  "lora_bias": false,
+  "lora_dropout": 0.05,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "peft_type": "LORA",
+  "peft_version": "0.18.1",
+  "qalora_group_size": 16,
+  "r": 8,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "mlp.c_proj",
+    "attn.c_attn",
+    "attn.c_proj",
+    "mlp.c_fc"
+  ],
+  "target_parameters": null,
+  "task_type": "CAUSAL_LM",
+  "trainable_token_indices": null,
+  "use_dora": false,
+  "use_qalora": false,
+  "use_rslora": false
+}

adapter_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1a44825e85a4250d4f3f938f4ba8f73e1630bebe8f36eb3395ccc390b9232853
+size 143610512

chat_template.jinja

@@ -0,0 +1,16 @@
+{% for message in messages %}
+{% if message['role'] == 'user' %}
+{{ 'Question:
+' + message['content'] + '
+
+' }}{% elif message['role'] == 'system' %}
+{{ 'System:
+' + message['content'] + '
+
+' }}{% elif message['role'] == 'assistant' %}{{ 'Answer:
+'  + message['content'] + '
+
+' }}{% endif %}
+{% if loop.last and add_generation_prompt %}
+{{ 'Answer:
+' }}{% endif %}{% endfor %}

checkpoint-159/README.md

@@ -0,0 +1,207 @@
+---
+base_model: ibm-granite/granite-20b-code-instruct-8k
+library_name: peft
+pipeline_tag: text-generation
+tags:
+- base_model:adapter:ibm-granite/granite-20b-code-instruct-8k
+- lora
+- transformers
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]
+### Framework versions
+
+- PEFT 0.18.1

checkpoint-159/adapter_config.json

@@ -0,0 +1,43 @@
+{
+  "alora_invocation_tokens": null,
+  "alpha_pattern": {},
+  "arrow_config": null,
+  "auto_mapping": null,
+  "base_model_name_or_path": "ibm-granite/granite-20b-code-instruct-8k",
+  "bias": "none",
+  "corda_config": null,
+  "ensure_weight_tying": false,
+  "eva_config": null,
+  "exclude_modules": null,
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 16,
+  "lora_bias": false,
+  "lora_dropout": 0.05,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "peft_type": "LORA",
+  "peft_version": "0.18.1",
+  "qalora_group_size": 16,
+  "r": 8,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "mlp.c_proj",
+    "attn.c_attn",
+    "attn.c_proj",
+    "mlp.c_fc"
+  ],
+  "target_parameters": null,
+  "task_type": "CAUSAL_LM",
+  "trainable_token_indices": null,
+  "use_dora": false,
+  "use_qalora": false,
+  "use_rslora": false
+}

checkpoint-159/adapter_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1a44825e85a4250d4f3f938f4ba8f73e1630bebe8f36eb3395ccc390b9232853
+size 143610512

checkpoint-159/chat_template.jinja

@@ -0,0 +1,16 @@
+{% for message in messages %}
+{% if message['role'] == 'user' %}
+{{ 'Question:
+' + message['content'] + '
+
+' }}{% elif message['role'] == 'system' %}
+{{ 'System:
+' + message['content'] + '
+
+' }}{% elif message['role'] == 'assistant' %}{{ 'Answer:
+'  + message['content'] + '
+
+' }}{% endif %}
+{% if loop.last and add_generation_prompt %}
+{{ 'Answer:
+' }}{% endif %}{% endfor %}

checkpoint-159/optimizer.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3ad37dd1d338c0ec168fceb829e7b90860de022b48008b734a5181e6b2441371
+size 73390503

checkpoint-159/rng_state.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:76124ad57cd60f8765ae756ad7354b12e1b65fc1c51aaedba29b59951d9667d4
+size 14645

checkpoint-159/scheduler.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b904ecdbc23198523387cccfa8a56e25f6eb34afce6e2c18607f88714579b221
+size 1465

checkpoint-159/special_tokens_map.json

@@ -0,0 +1,45 @@
+{
+  "additional_special_tokens": [
+    "<|endoftext|>",
+    "<fim_prefix>",
+    "<fim_middle>",
+    "<fim_suffix>",
+    "<fim_pad>",
+    "<filename>",
+    "<gh_stars>",
+    "<issue_start>",
+    "<issue_comment>",
+    "<issue_closed>",
+    "<jupyter_start>",
+    "<jupyter_text>",
+    "<jupyter_code>",
+    "<jupyter_output>",
+    "<empty_output>",
+    "<commit_before>",
+    "<commit_msg>",
+    "<commit_after>",
+    "<reponame>"
+  ],
+  "bos_token": {
+    "content": "<|endoftext|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eos_token": {
+    "content": "<|endoftext|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": "<|endoftext|>",
+  "unk_token": {
+    "content": "<|endoftext|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

checkpoint-159/tokenizer_config.json

@@ -0,0 +1,188 @@
+{
+  "add_prefix_space": false,
+  "added_tokens_decoder": {
+    "0": {
+      "content": "<|endoftext|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "1": {
+      "content": "<fim_prefix>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "2": {
+      "content": "<fim_middle>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "3": {
+      "content": "<fim_suffix>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "4": {
+      "content": "<fim_pad>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "5": {
+      "content": "<filename>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "6": {
+      "content": "<gh_stars>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "7": {
+      "content": "<issue_start>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "8": {
+      "content": "<issue_comment>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "9": {
+      "content": "<issue_closed>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "10": {
+      "content": "<jupyter_start>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "11": {
+      "content": "<jupyter_text>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "12": {
+      "content": "<jupyter_code>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "13": {
+      "content": "<jupyter_output>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "14": {
+      "content": "<empty_output>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "15": {
+      "content": "<commit_before>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "16": {
+      "content": "<commit_msg>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "17": {
+      "content": "<commit_after>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "18": {
+      "content": "<reponame>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "additional_special_tokens": [
+    "<|endoftext|>",
+    "<fim_prefix>",
+    "<fim_middle>",
+    "<fim_suffix>",
+    "<fim_pad>",
+    "<filename>",
+    "<gh_stars>",
+    "<issue_start>",
+    "<issue_comment>",
+    "<issue_closed>",
+    "<jupyter_start>",
+    "<jupyter_text>",
+    "<jupyter_code>",
+    "<jupyter_output>",
+    "<empty_output>",
+    "<commit_before>",
+    "<commit_msg>",
+    "<commit_after>",
+    "<reponame>"
+  ],
+  "bos_token": "<|endoftext|>",
+  "clean_up_tokenization_spaces": true,
+  "eos_token": "<|endoftext|>",
+  "extra_special_tokens": {},
+  "model_max_length": 8192,
+  "pad_token": "<|endoftext|>",
+  "padding_side": "right",
+  "tokenizer_class": "GPT2Tokenizer",
+  "unk_token": "<|endoftext|>",
+  "vocab_size": 49152
+}

checkpoint-159/trainer_state.json

@@ -0,0 +1,139 @@
+{
+  "best_global_step": null,
+  "best_metric": null,
+  "best_model_checkpoint": null,
+  "epoch": 3.0,
+  "eval_steps": 500,
+  "global_step": 159,
+  "is_hyper_param_search": false,
+  "is_local_process_zero": true,
+  "is_world_process_zero": true,
+  "log_history": [
+    {
+      "epoch": 0.1902497027348395,
+      "grad_norm": 1.049843430519104,
+      "learning_rate": 1.8e-05,
+      "loss": 3.6094,
+      "step": 10
+    },
+    {
+      "epoch": 0.380499405469679,
+      "grad_norm": 1.4530141353607178,
+      "learning_rate": 3.8e-05,
+      "loss": 3.3156,
+      "step": 20
+    },
+    {
+      "epoch": 0.5707491082045184,
+      "grad_norm": 1.1667392253875732,
+      "learning_rate": 5.8e-05,
+      "loss": 2.6267,
+      "step": 30
+    },
+    {
+      "epoch": 0.760998810939358,
+      "grad_norm": 1.4776651859283447,
+      "learning_rate": 7.800000000000001e-05,
+      "loss": 2.1165,
+      "step": 40
+    },
+    {
+      "epoch": 0.9512485136741974,
+      "grad_norm": 1.4660998582839966,
+      "learning_rate": 9.8e-05,
+      "loss": 1.804,
+      "step": 50
+    },
+    {
+      "epoch": 1.1331747919143877,
+      "grad_norm": 1.163648247718811,
+      "learning_rate": 0.000118,
+      "loss": 1.5531,
+      "step": 60
+    },
+    {
+      "epoch": 1.323424494649227,
+      "grad_norm": 1.323768973350525,
+      "learning_rate": 0.000138,
+      "loss": 1.4815,
+      "step": 70
+    },
+    {
+      "epoch": 1.5136741973840666,
+      "grad_norm": 1.6716187000274658,
+      "learning_rate": 0.00015800000000000002,
+      "loss": 1.3993,
+      "step": 80
+    },
+    {
+      "epoch": 1.7039239001189062,
+      "grad_norm": 1.709359049797058,
+      "learning_rate": 0.00017800000000000002,
+      "loss": 1.3135,
+      "step": 90
+    },
+    {
+      "epoch": 1.8941736028537455,
+      "grad_norm": 1.735700249671936,
+      "learning_rate": 0.00019800000000000002,
+      "loss": 1.2533,
+      "step": 100
+    },
+    {
+      "epoch": 2.0760998810939357,
+      "grad_norm": 1.7600226402282715,
+      "learning_rate": 0.00018873520750565718,
+      "loss": 1.1572,
+      "step": 110
+    },
+    {
+      "epoch": 2.2663495838287755,
+      "grad_norm": 2.0459885597229004,
+      "learning_rate": 0.00015304209081197425,
+      "loss": 0.9394,
+      "step": 120
+    },
+    {
+      "epoch": 2.456599286563615,
+      "grad_norm": 1.884280800819397,
+      "learning_rate": 0.00010266205214377748,
+      "loss": 0.9113,
+      "step": 130
+    },
+    {
+      "epoch": 2.646848989298454,
+      "grad_norm": 2.249265432357788,
+      "learning_rate": 5.1544912966734994e-05,
+      "loss": 0.8832,
+      "step": 140
+    },
+    {
+      "epoch": 2.837098692033294,
+      "grad_norm": 2.0327601432800293,
+      "learning_rate": 1.3844591860619383e-05,
+      "loss": 0.9004,
+      "step": 150
+    }
+  ],
+  "logging_steps": 10,
+  "max_steps": 159,
+  "num_input_tokens_seen": 0,
+  "num_train_epochs": 3,
+  "save_steps": 500,
+  "stateful_callbacks": {
+    "TrainerControl": {
+      "args": {
+        "should_epoch_stop": false,
+        "should_evaluate": false,
+        "should_log": false,
+        "should_save": true,
+        "should_training_stop": true
+      },
+      "attributes": {}
+    }
+  },
+  "total_flos": 6.123192092794552e+17,
+  "train_batch_size": 1,
+  "trial_name": null,
+  "trial_params": null
+}

checkpoint-159/training_args.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:8235488be59bf3fd0161c7b433cebdef0a21ee6ed0d25ce0a1eed891f0042f8f
+size 5905

special_tokens_map.json

@@ -0,0 +1,45 @@
+{
+  "additional_special_tokens": [
+    "<|endoftext|>",
+    "<fim_prefix>",
+    "<fim_middle>",
+    "<fim_suffix>",
+    "<fim_pad>",
+    "<filename>",
+    "<gh_stars>",
+    "<issue_start>",
+    "<issue_comment>",
+    "<issue_closed>",
+    "<jupyter_start>",
+    "<jupyter_text>",
+    "<jupyter_code>",
+    "<jupyter_output>",
+    "<empty_output>",
+    "<commit_before>",
+    "<commit_msg>",
+    "<commit_after>",
+    "<reponame>"
+  ],
+  "bos_token": {
+    "content": "<|endoftext|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eos_token": {
+    "content": "<|endoftext|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": "<|endoftext|>",
+  "unk_token": {
+    "content": "<|endoftext|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

tokenizer_config.json

@@ -0,0 +1,188 @@
+{
+  "add_prefix_space": false,
+  "added_tokens_decoder": {
+    "0": {
+      "content": "<|endoftext|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "1": {
+      "content": "<fim_prefix>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "2": {
+      "content": "<fim_middle>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "3": {
+      "content": "<fim_suffix>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "4": {
+      "content": "<fim_pad>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "5": {
+      "content": "<filename>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "6": {
+      "content": "<gh_stars>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "7": {
+      "content": "<issue_start>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "8": {
+      "content": "<issue_comment>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "9": {
+      "content": "<issue_closed>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "10": {
+      "content": "<jupyter_start>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "11": {
+      "content": "<jupyter_text>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "12": {
+      "content": "<jupyter_code>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "13": {
+      "content": "<jupyter_output>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "14": {
+      "content": "<empty_output>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "15": {
+      "content": "<commit_before>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "16": {
+      "content": "<commit_msg>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "17": {
+      "content": "<commit_after>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "18": {
+      "content": "<reponame>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "additional_special_tokens": [
+    "<|endoftext|>",
+    "<fim_prefix>",
+    "<fim_middle>",
+    "<fim_suffix>",
+    "<fim_pad>",
+    "<filename>",
+    "<gh_stars>",
+    "<issue_start>",
+    "<issue_comment>",
+    "<issue_closed>",
+    "<jupyter_start>",
+    "<jupyter_text>",
+    "<jupyter_code>",
+    "<jupyter_output>",
+    "<empty_output>",
+    "<commit_before>",
+    "<commit_msg>",
+    "<commit_after>",
+    "<reponame>"
+  ],
+  "bos_token": "<|endoftext|>",
+  "clean_up_tokenization_spaces": true,
+  "eos_token": "<|endoftext|>",
+  "extra_special_tokens": {},
+  "model_max_length": 8192,
+  "pad_token": "<|endoftext|>",
+  "padding_side": "right",
+  "tokenizer_class": "GPT2Tokenizer",
+  "unk_token": "<|endoftext|>",
+  "vocab_size": 49152
+}

training_args.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:8235488be59bf3fd0161c7b433cebdef0a21ee6ed0d25ce0a1eed891f0042f8f
+size 5905