secret-scanner-tool 1.0.0
A lightweight, extensible tool to automatically scan source code for hardcoded secrets and PII
License: MIT
Requires Python: >=3.7
Keywords: security, secrets, pii, detection, scanning, git, pre-commit, ci-cd, static-analysis, code-review

# Secret & PII Scanner

This tool detects hardcoded secrets and PII in source code using pattern-based (regex) detection only.

## Features
- Detects API keys, tokens, passwords, emails, credit cards, and more using regex patterns.
- Customizable detection patterns via `.secret-scanner.yaml`.
- Multi-language support.
- Rich reporting (HTML, JSON, etc.).

## How It Works
- The scanner uses a set of regex patterns to match secrets and PII in your codebase.
- No entropy-based detection is used; only explicit pattern matches are reported.

## Configuration
- Add or customize patterns in `.secret-scanner.yaml`.

---

## 🚀 Quick Start

```bash
# 1. Install (from source, recommended for latest features)
git clone https://gitlab.com/ox-saro/SecretDetection.git
cd SecretDetection
pip install -e .

# 2. Run your first scan
secret-scanner scan .

# 3. See all options
secret-scanner --help
```

---

## 📦 Installation

### System Requirements
- **Python**: 3.7 or higher
- **Git**: For diff-based scanning and pre-commit hooks
- **Operating Systems**: Windows, macOS, Linux

### Check System Compatibility
Before installing, verify your system compatibility:
```bash
# After installation, run this to check your system
secret-scanner check-system
```

### From Source (Recommended)
```bash
git clone https://gitlab.com/ox-saro/SecretDetection.git
cd SecretDetection
pip install -e .
```

### From Built Package
```bash
python3 build_package.py build
pip install dist/secret_scanner-1.0.0-py3-none-any.whl
```

### For Development
```bash
python -m venv venv
source venv/bin/activate  # On Windows: venv\Scripts\activate
pip install -e ".[dev]"
```

### Platform-Specific Installation

#### Windows
```bash
# Install Python 3.7+ from https://python.org
# Install Git from https://git-scm.com
pip install secret-scanner[pdf]
```

#### macOS
```bash
# Using Homebrew (recommended)
brew install python3 git
pip3 install secret-scanner[pdf]

# For PDF support with WeasyPrint
brew install cairo pango gdk-pixbuf libffi
pip3 install weasyprint
```

#### Linux (Ubuntu/Debian)
```bash
sudo apt-get update
sudo apt-get install python3 python3-pip git
pip3 install secret-scanner[pdf]

# For PDF support with WeasyPrint
sudo apt-get install build-essential python3-dev python3-pip python3-setuptools python3-wheel python3-cffi libcairo2 libpango-1.0-0 libpangocairo-1.0-0 libgdk-pixbuf2.0-0 libffi-dev shared-mime-info
pip3 install weasyprint
```

#### Linux (CentOS/RHEL)
```bash
sudo yum install python3 python3-pip git
pip3 install secret-scanner[pdf]

# For PDF support with WeasyPrint
sudo yum install gcc python3-devel cairo-devel pango-devel gdk-pixbuf2-devel libffi-devel
pip3 install weasyprint
```

---

## 🏃 Usage

### CLI (Recommended)

```bash
# Scan a directory
secret-scanner scan /path/to/repo

# Scan only Python and JS files
secret-scanner scan . --files "*.py,*.js"

# Scan only changed files (git diff)
secret-scanner scan . --diff-only

# Output as JSON, HTML, or PDF with custom path
secret-scanner scan . --output json --report-path /tmp/my_report.json
secret-scanner scan . --output html --report-name custom_report.html
secret-scanner scan . --output pdf --report-path /reports/security_scan.pdf

# Multi-threaded scan with progress tracking
secret-scanner scan . --max-workers 8 --chunk-size 20 --verbose

# Multi-project scan with custom report
secret-scanner scan-multi /path/to/projects --report-name multi_scan_summary.html

# Install pre-commit hook
secret-scanner install-hook

# Test a regex pattern
secret-scanner test "api_key_[a-zA-Z0-9]{32}" "api_key_1234567890abcdef1234567890abcdef"
```

### As a Python Module

```python
from secret_scanner.core.scanner import SecretScanner
from secret_scanner.core.config import Config

config = Config.load_default()
scanner = SecretScanner(config)
result = scanner.scan_directory("/path/to/repo")
print(f"Found {len(result.findings)} secrets")
```

---

## ⚙️ Configuration

Create `.secret-scanner.yaml` in your repo root:

```yaml
patterns:
  - name: "Custom API Key"
    regex: "my_custom_key_[a-zA-Z0-9]{32}"
    severity: "high"

exclude:
  - "*.log"
  - "node_modules/"
  - "vendor/"

max_file_size: 10485760  # 10MB
context_lines: 3
max_workers: 8
chunk_size: 10
```

- **Custom Patterns**: Add your own regex rules
- **Exclude**: Skip files or directories
- **Multi-threading**: Tune `max_workers` and `chunk_size`

---

## 📧 Email Notification Setup

1. **Create `email_config.json`:**
```json
{
  "smtp_server": "smtp.gmail.com",
  "smtp_port": 587,
  "username": "your-email@gmail.com",
  "password": "your-app-password",
  "use_tls": true,
  "from_email": "your-email@gmail.com",
  "to_emails": ["recipient@example.com"]
}
```

2. **Test Email Setup:**
```bash
secret-scanner scan-multi . --test-email --email-config email_config.json
```

3. **Send Reports via Email:**
```bash
secret-scanner scan-multi /path/to/projects --send-email --email-config email_config.json
```

- **Gmail**: Use an App Password (see Google Account > Security)
- **Other Providers**: Use your SMTP settings

---

## 🏗️ Build & Run

### Build the Package
```bash
python3 build_package.py build
```

### Install from Build
```bash
pip install dist/secret_scanner-1.0.0-py3-none-any.whl
```

### Run the CLI
```bash
secret-scanner scan . --help
```

---

## 🆕 Enhanced Features

### Real-time Progress Tracking
The scanner now provides beautiful progress bars showing:
- Current file being scanned
- Number of findings discovered
- Completion percentage and time elapsed
- Multi-project scan progress with project-level updates

```bash
# Watch progress in real-time
secret-scanner scan . --verbose
```

### Custom Report Paths
Specify exactly where and how to save your reports:

```bash
# Full path (overrides report name)
secret-scanner scan . --output json --report-path /tmp/custom_report.json

# Custom filename in current directory
secret-scanner scan . --output html --report-name my_scan_report.html

# Multi-project with custom summary report
secret-scanner scan-multi /projects --report-name security_summary.pdf
```

### Improved Pattern Accuracy
Enhanced regex patterns reduce false positives:
- Most patterns now require assignment operators (`=`, `:`) or context
- Better detection of actual secrets vs. legitimate text
- Improved API key and token pattern matching
- More precise PII detection

---

## 🛠️ Troubleshooting

- **Missing command?**
  - Reinstall: `pip install -e .`
  - Check: `which secret-scanner`
- **Import errors?**
  - Check: `python -c "import secret_scanner; print(secret_scanner.__version__)"`
- **PDF/email issues?**
  - Ensure system dependencies for WeasyPrint are installed (see their docs)
- **Email not sending?**
  - Check SMTP credentials and provider restrictions
- **Performance?**
  - Use multi-threading: `--max-workers 8 --chunk-size 20`
  - Exclude large/unnecessary directories

---

## 🤝 Contributing

1. Fork the repo
2. Create a feature branch
3. Add your changes & tests
4. Submit a pull request

- Code style: [Black](https://black.readthedocs.io/), [flake8](https://flake8.pycqa.org/)
- Tests: [pytest](https://docs.pytest.org/)

---

## 📄 License

MIT License. See [LICENSE](LICENSE) for details.

---

## 🆘 Support

- Create an issue for bugs or questions
- Check this README for common solutions

---

## 🕒 Version History

- **v1.0.0** (Latest): Initial release with multi-threaded scanning, 50+ patterns, multi-project/email support, rich CLI, progress tracking, custom report paths, and improved regex accuracy

---