veritensor 1.5.1
Antivirus for the AI Supply Chain. Scans models, datasets, notebooks, and RAG documents for threats.
License: Apache-2.0
Requires Python: >=3.10
Keywords: security, ai, mlops, malware-detection, supply-chain, devsecops, rag, data-poisoning

# 🛡️ Veritensor: AI Data & Artifact Security

[![Hugging Face Spaces](https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Spaces-blue)](https://huggingface.co/spaces/arsbr/veritensor-ai-model-security-scanner)
[![PyPI version](https://img.shields.io/pypi/v/veritensor?color=blue&logo=pypi&logoColor=white)](https://pypi.org/project/veritensor/)
[![Docker Image](https://img.shields.io/docker/v/arseniibrazhnyk/veritensor?label=docker&color=blue&logo=docker&logoColor=white)](https://hub.docker.com/r/arseniibrazhnyk/veritensor)
[![License](https://img.shields.io/badge/License-Apache%202.0-green.svg)](https://opensource.org/licenses/Apache-2.0)
[![CI](https://github.com/arsbr/Veritensor/actions/workflows/scanner-ci.yaml/badge.svg)](https://github.com/arsbr/Veritensor/actions/workflows/scanner-ci.yaml)
[![Security](https://github.com/arsbr/Veritensor/actions/workflows/security.yaml/badge.svg)](https://github.com/arsbr/Veritensor/actions/workflows/security.yaml)
[![Security: Veritensor](https://img.shields.io/badge/Security-Veritensor-0096FF?style=flat&logo=security&logoColor=white)](https://github.com/arsbr/veritensor)

**Veritensor** is an end-to-end antivirus for the entire AI Life Cycle. It secures the entire AI Supply Chain by scanning artifacts that traditional tools miss: Models, Datasets, RAG Documents, and Notebooks.

Unlike standard SAST tools (which focus on code), Veritensor understands the binary and serialized formats used in Machine Learning:
1.  **Models:** Deep AST analysis of **Pickle, PyTorch, Keras, Safetensors** to block RCE and backdoors.
2.  **Data & RAG:** Streaming scan of **Parquet, CSV, Excel, PDF** to detect Data Poisoning, Prompt Injections, and PII.
3.  **Notebooks:** Hardening of **Jupyter (.ipynb)** files by detecting leaked secrets (using Entropy analysis), malicious magics, and XSS.
4.  **Supply Chain:** Audits **dependencies** (`requirements.txt`, `poetry.lock`) for Typosquatting and known CVEs (via OSV.dev).
5.  **Governance:** Generates cryptographic **Data Manifests** (Provenance) and signs containers via **Sigstore**.

---

## 🚀 Features

*   **Parallel Scanning:** Utilizes all CPU cores to scan thousands of files in seconds. Includes robust **SQLite Caching** to skip unchanged files.
*   **Stealth Detection:** Finds attacks hidden from humans but visible to LLMs. Detects **CSS Hiding** (white text, zero font), **Base64 Obfuscation**, and **Unicode Spoofing**.
*   **Dataset Security:** Streams massive datasets (100GB+) to find "Poisoning" patterns (e.g., "Ignore previous instructions") and malicious URLs in **Parquet, CSV, JSONL, and Excel**.
*   **Archive Inspection:** Safely scans inside **.zip, .tar.gz, .whl** files without extracting them to disk (Zip Bomb protected).
*   **Dependency Audit:** Checks `pyproject.toml`, `poetry.lock`, and `Pipfile.lock` for malicious packages (Typosquatting) and vulnerabilities.
*   **Data Provenance:** Command `veritensor manifest .` creates a signed JSON snapshot of your data artifacts for compliance (EU AI Act).
*   **Identity Verification:** Automatically verifies model hashes against the official Hugging Face registry to detect Man-in-the-Middle attacks.

---

## 📦 Installation

Veritensor is modular. Install only what you need to keep your environment lightweight (~50MB core).

| Option | Command | Use Case |
| :--- | :--- | :--- |
| **Core** | `pip install veritensor` | Base scanner (Models, Notebooks, Dependencies) |
| **Data** | `pip install "veritensor[data]"` | Datasets (Parquet, Excel, CSV) |
| **RAG** | `pip install "veritensor[rag]"` | Documents (PDF, DOCX, PPTX) |
| **PII** | `pip install "veritensor[pii]"` | ML-based PII detection (Presidio) |
| **AWS** | `pip install "veritensor[aws]"` | Direct scanning from S3 buckets |
| **All** | `pip install "veritensor[all]"` | Full suite for enterprise security |

### Via Docker (Recommended for CI/CD)
```bash
docker pull arseniibrazhnyk/veritensor:latest
```

---

## ⚡ Quick Start

### 1. Scan a local project (Parallel)
Recursively scan a directory for all supported threats using 4 CPU cores:
```bash
veritensor scan ./my-rag-project --recursive --jobs 4
```

### 2. Scan RAG Documents & Excel
Check for Prompt Injections and Formula Injections in business data:
```bash
veritensor scan ./finance_data.xlsx
veritensor scan ./docs/contract.pdf
```

### 3. Generate Data Manifest
Create a compliance snapshot of your dataset folder:
```bash
veritensor manifest ./data --output provenance.json
```

### 4. Verify Model Integrity
Ensure the file on your disk matches the official version from Hugging Face (detects tampering):
```bash
veritensor scan ./pytorch_model.bin --repo meta-llama/Llama-2-7b
```

### 5. Scan from Amazon S3
Scan remote assets without manual downloading:
```bash
veritensor scan s3://my-ml-bucket/models/llama-3.pkl
```

### 6. Verify against Hugging Face
Ensure the file on your disk matches the official version from the registry (detects tampering):
```bash
veritensor scan ./pytorch_model.bin --repo meta-llama/Llama-2-7b
```

### 7. License Compliance Check
Veritensor automatically reads metadata from safetensors and GGUF files.
If a model has a Non-Commercial license (e.g., cc-by-nc-4.0), it will raise a HIGH severity alert.

To override this (Break-glass mode), use:
```bash
veritensor scan ./model.safetensors --force
```

### 8. Scan AI Datasets
Veritensor uses streaming to handle huge files. It samples 10k rows by default for speed.
```bash
veritensor scan ./data/train.parquet --full-scan
```

### 9. Scan Jupyter Notebooks
Check code cells, markdown, and saved outputs for threats:
```bash
veritensor scan ./research/experiment.ipynb
```

**Example Output:**
```Text
╭────────────────────────────────╮
│ 🛡️  Veritensor Security Scanner │
╰────────────────────────────────╯
                                    Scan Results
┏━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━┓
┃ File         ┃ Status ┃ Threats / Details                    ┃ SHA256 (Short) ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━┩
│ model.pt     │  FAIL  │ CRITICAL: os.system (RCE Detected)   │ a1b2c3d4...    │
└──────────────┴────────┴──────────────────────────────────────┴────────────────┘
❌ BLOCKING DEPLOYMENT
```

---

## 📊 Reporting & Compliance

Veritensor supports industry-standard formats for integration with security dashboards and audit tools.

### 1. GitHub Security (SARIF)
Generate a report compatible with GitHub Code Scanning:
```bash
veritensor scan ./models --sarif > veritensor-report.sarif
```
### 2. Software Bill of Materials (SBOM)
Generate a CycloneDX v1.5 SBOM to inventory your AI assets:
```bash
veritensor scan ./models --sbom > sbom.json
```
### 3. Raw JSON
For custom parsers and SOAR automation:
```bash
veritensor scan ./models --json
```

---

## 🔐 Supply Chain Security (Container Signing)

Veritensor integrates with Sigstore Cosign to cryptographically sign your Docker images only if they pass the security scan.

### 1. Generate Keys
Generate a key pair for signing:
```bash
veritensor keygen
# Output: veritensor.key (Private) and veritensor.pub (Public)
```
### 2. Scan & Sign
Pass the --image flag and the path to your private key (via env var).
```bash
# Set path to your private key
export VERITENSOR_PRIVATE_KEY_PATH=veritensor.key

# If scan passes -> Sign the image
veritensor scan ./models/my_model.pkl --image my-org/my-app:v1.0.0
```
### 3. Verify (In Kubernetes / Production)
Before deploying, verify the signature to ensure the model was scanned:
```bash
cosign verify --key veritensor.pub my-org/my-app:v1.0.0
```

---

## 🛠️ Integrations

### GitHub Actions
Add this to your .github/workflows/security.yml to block malicious models in Pull Requests:
```yaml
name: AI Security Scan
on: [pull_request]
jobs:
  veritensor-scan:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Veritensor Scan
        uses: ArseniiBrazhnyk/Veritensor@v1.5.1
        with:
          path: '.'
          args: '--jobs 4'
```
### Pre-commit Hook

Prevent committing malicious models to your repository. Add this to .pre-commit-config.yaml:
```yaml
repos:
  - repo: https://github.com/arsbr/Veritensor
    rev: v1.5.1
    hooks:
      - id: veritensor-scan
```

---

## 📂 Supported Formats

| Format | Extension | Analysis Method |
| :--- | :--- | :--- |
| **Models** | `.pt`, `.pth`, `.bin`, `.pkl`, `.joblib`, `.h5`, `.keras`, `.safetensors`, `.gguf`, `.whl`  | AST Analysis, Pickle VM Emulation, Metadata Validation |
| **Datasets** | `.parquet`, `.csv`, `.tsv`, `.jsonl`, `.ndjson`, `.ldjson` | Streaming Regex Scan (URLs, Injections, PII) |
| **Notebooks** | `.ipynb` | JSON Structure Analysis + Code AST + Markdown Phishing |
| **Documents** | `.pdf`, `.docx`, `.pptx`, `.txt`, `.md`, `.html` | DOM Extraction, Stealth/CSS Detection, PII |
| **Archives** | `.zip`, `.tar`, `.gz`, `.tgz`, `.whl` | Recursive In-Memory Inspection |
| **RAG Docs** | `requirements.txt`, `poetry.lock`, `Pipfile.lock` | Typosquatting, OSV.dev CVE Lookup |

---

## ⚙️ Configuration

You can customize security policies by creating a `veritensor.yaml` file in your project root.
Pro Tip: You can use `regex:` prefix for flexible matching.

```yaml
# veritensor.yaml

# 1. Security Threshold
# Fail the build if threats of this severity (or higher) are found.
# Options: CRITICAL, HIGH, MEDIUM, LOW.
fail_on_severity: CRITICAL

# 2. Dataset Scanning
# Sampling limit for quick scans (default: 10000)
dataset_sampling_limit: 10000

# 3. License Firewall Policy
# If true, blocks models that have no license metadata.
fail_on_missing_license: false

# List of license keywords to block (case-insensitive).
custom_restricted_licenses:
  - "cc-by-nc"       # Non-Commercial
  - "agpl"           # Viral licenses
  - "research-only"

# 4. Static Analysis Exceptions (Pickle)
# Allow specific Python modules that are usually blocked by the strict scanner.
allowed_modules:
  - "my_company.internal_layer"
  - "sklearn.tree"

# 5. Model Whitelist (License Bypass)
# List of Repo IDs that are trusted. Veritensor will SKIP license checks for these.
# Supports Regex!
allowed_models:
  - "meta-llama/Meta-Llama-3-70B-Instruct"  # Exact match
  - "regex:^google-bert/.*"                 # Allow all BERT models from Google
  - "internal/my-private-model"
```

To generate a default configuration file, run: veritensor init

---

## 🧠 Threat Intelligence (Signatures)

Veritensor uses a decoupled signature database (`signatures.yaml`) to detect malicious patterns. This ensures that detection logic is separated from the core engine.

*   **Automatic Updates:** To get the latest threat definitions, simply upgrade the package:
    ```bash
    pip install --upgrade veritensor
    ```
*   **Transparent Rules:** You can inspect the default signatures in `src/veritensor/engines/static/signatures.yaml`.
*   **Custom Policies:** If the default rules are too strict for your use case (false positives), use `veritensor.yaml` to whitelist specific modules or models.

  ---

## 📜 License

This project is licensed under the Apache 2.0 License - see the [LICENSE](https://github.com/arsbr/Veritensor?tab=Apache-2.0-1-ov-file#readme) file for details.