norm-findings 0.4.3
A lean CLI tool for normalizing security scanner findings based on DefectDojo parsers.
License: Apache-2.0
Requires Python: >=3.9

# norm-findings
A lean CLI tool for normalizing security scanner findings based on DefectDojo parsers.

This project provides a standalone Python package and a minimal Docker image to convert findings from O(100) security scanners into a normalized format.

## Open Source Attribution
This project is based on the excellent work of the [DefectDojo](https://github.com/DefectDojo/django-DefectDojo) community. We leverage their parser logic while providing a lean, dependency-minimized execution environment. See the [NOTICE](NOTICE) file for more details.

## Installation

The default installation includes the core CLI and **all parser dependencies**, providing full functionality out-of-the-box.

### Standard (Core + Parsers)
```bash
pip install .
```

### Optional: Server Support
If you need the REST API server, install the server extra:
```bash
pip install ".[server]"
```

### Optional: Development
For running tests or contributing:
```bash
pip install ".[dev]"
```

## Running Tests

### Unit Tests
Verify the core installation and stubs:
```bash
pytest tests/test_cli.py
```

### E2E Parser Verification (Development only)
To verify all 200+ parsers against real DefectDojo sample data:
1. Ensure the development dependencies are installed (`pip install ".[dev]"`).
2. Run the updater to fetch sample data:
   ```bash
   python -m norm_findings.updater
   ```
3. Run the E2E tests:
   ```bash
   pytest tests/test_e2e.py
   ```

## Usage

`norm-findings` supports **200+ security scanners** and accepts parser names in a **case-insensitive** manner, with or without the "Parser" suffix.

### CLI

The CLI supports all parsers generically - just specify the parser name (case-insensitive):

```bash
# Basic usage - parser names are case-insensitive
norm-findings convert --parser trivy --input-file trivy.json --output-file findings.json

# Works with any case
norm-findings convert --parser TRIVY --input-file trivy.json --output-file findings.json
norm-findings convert --parser Trivy --input-file trivy.json --output-file findings.json

# Works with or without "Parser" suffix
norm-findings convert --parser TrivyParser --input-file trivy.json --output-file findings.json
norm-findings convert --parser anchore_grype --input-file grype.json --output-file findings.json

# With custom test label
norm-findings convert --parser bandit --input-file bandit.json --output-file findings.json --test "My App v1.0"

# Print findings to console
norm-findings convert --parser semgrep --input-file semgrep.json --output-file findings.json --print
```

**Available parsers** (200+): `trivy`, `anchore_grype`, `bandit`, `semgrep`, `snyk`, `checkmarx`, `fortify`, `zap`, `burp`, `aqua`, `blackduck`, `sonarqube`, `veracode`, and many more. See [parser_mapping.json](norm_findings/parser_mapping.json) for the complete list.

### Docker

```bash
# Case-insensitive parser names work in Docker too
docker run -v $(pwd):/dojo -it ghcr.io/scribe-security/norm-findings:latest \
  convert --parser trivy --input-file /dojo/trivy.json --output-file /dojo/findings.json

docker run -v $(pwd):/dojo -it ghcr.io/scribe-security/norm-findings:latest \
  convert --parser ANCHORE_GRYPE --input-file /dojo/grype.json --output-file /dojo/findings.json
```

### Using as a Library (API)

The recommended way to use `norm-findings` as a library is through the high-level API, which handles parser resolution automatically:

```python
from norm_findings.api import parse_findings

# Parse findings - parser name is case-insensitive
findings = parse_findings(
    parser_name="trivy",  # Case-insensitive: "trivy", "TRIVY", "Trivy" all work
    input_file="trivy.json",
    test_label="My Application v1.0",
    output_file="findings.json"  # Optional: writes findings to file
)

# Process findings
for finding in findings:
    print(f"{finding.severity}: {finding.title}")
    if finding.description:
        print(f"  Description: {finding.description}")
```

**More API Examples:**

```python
# Example 1: Parse without writing to file
findings = parse_findings(
    parser_name="anchore_grype",
    input_file="grype-results.json",
    test_label="Production Scan"
)
print(f"Found {len(findings)} vulnerabilities")

# Example 2: Parsing with Robustness (Capture Errors)
# For robustness, parsers log errors instead of crashing. 
# You can capture these errors using return_details=True.
result = parse_findings(
    parser_name="semgrep",
    input_file="semgrep_report.json",
    return_details=True
)

findings = result['findings']
errors = result['errors']

if not result['success']:
    print(f"Warning: Parsed {len(findings)} findings but encountered {len(errors)} errors:")
    for error in errors:
        print(error)

# Example 3: Error handling (Invalid Parser/File)
try:
    findings = parse_findings(
        parser_name="my_scanner",
        input_file="results.json"
    )
except ValueError as e:
    print(f"Parser error: {e}")
```

**Advanced: Direct Parser Usage**

If you need more control, you can instantiate parsers directly:

```python
from norm_findings.parsers.trivy.parser import TrivyParser
from norm_findings.stubs.models import Test

parser = TrivyParser()
test = Test(product="My Application")

with open("trivy.json", "rb") as f:
    findings = parser.get_findings(f, test)

for finding in findings:
    print(f"Found: {finding.title} ({finding.severity})")
```

## Legacy Version
The original monkey-patched version of this tool is preserved in the `legacy-monkeypatch` branch and tagged as `v1.x-legacy`. 

To use the legacy version:
```bash
git checkout v1.x-legacy
```

## Automatic Updates
`norm-findings` includes a built-in updater that fetches the latest parsers and tests from DefectDojo:
```bash
python -m norm_findings.updater
```

## Development

### Workflow
1.  **Branching**: Create a new branch for your feature or bugfix from `main`.
2.  **Syncing Parsers**: Run the updater to ensure you have the latest DefectDojo parsers:
    ```bash
    python -m norm_findings.updater
    ```
3.  **Testing**: Always run the test suite before pushing:
    ```bash
    pytest tests/test_cli.py
    pytest tests/test_e2e.py --ignore norm_findings/stubs/models.py
    ```
4.  **Pushing**: Push your branch to GitHub and open a Pull Request.

### Versioning
`norm-findings` uses [setuptools-scm](https://github.com/pypa/setuptools-scm) for automatic versioning. 
- The version is automatically derived from the most recent Git tag.
- When working on local uncommitted changes, the version will include a `.dev` suffix and the current timestamp.
- The version is written to `norm_findings/_version.py` during the build process.

### Releasing
Releases are automated via GitHub Actions and are triggered by pushing a version tag:

1.  **Create a tag**: Create a semantic version tag starting with `v` (e.g., `v1.1.0`):
    ```bash
    git tag -a v1.1.0 -m "Release version 1.1.0"
    ```
2.  **Push the tag**:
    ```bash
    git push origin v1.1.0
    ```
3.  **Automated Pipeline**: The [build workflow](.github/workflows/build.yml) will automatically:
    - Run all tests.
    - Build the Python wheel and source distribution.
    - Publish to **PyPI**.
    - Build and push the Docker image to **GHCR** (tagged with the version and `latest`).

### Automatic Parser Updates
A daily [GitHub Action](.github/workflows/check-updates.yml) runs the `updater.py` logic. If new parsers or updates are detected in DefectDojo:
1.  A new branch `auto-update-parsers` is created.
2.  A Pull Request is opened with a summary of the changes.
3.  Maintainers can review and merge the PR to keep `norm-findings` up-to-date.