ai-testing-swarm 0.1.17
AI-powered testing swarm
License: MIT
Requires Python: >=3.9

# AI Testing Swarm

AI Testing Swarm is a **super-advanced, mutation-driven API testing framework** (with optional OpenAPI + OpenAI augmentation) built on top of **pytest**.

It generates a large set of deterministic negative/edge/security test cases for an API request, executes them (optionally in parallel, with retries/throttling), and produces a report (JSON/Markdown/HTML) with summaries.

> Notes:
> - UI testing is not the focus of the current releases.
> - OpenAI features are **optional** and disabled by default.

---

## Installation

```bash
pip install ai-testing-swarm
```

Optional (OpenAPI JSON schema validation for responses):

```bash
pip install "ai-testing-swarm[openapi]"
```

CLI entrypoint:

```bash
ai-test --help
```

---

## Quick start (cURL input)

Create `request.json`:

```json
{
  "curl": "curl --location https://postman-echo.com/post --header \"Content-Type: application/json\" --data \"{\\\"hello\\\":\\\"world\\\",\\\"count\\\":1}\""
}
```

Run:

```bash
ai-test --input request.json
```

Choose a report format:

```bash
ai-test --input request.json --report-format html
```

A report is written under:

- `./ai_swarm_reports/<METHOD>_<endpoint>/<METHOD>_<endpoint>_<timestamp>.<json|md|html>`

Reports include:
- per-test results (including deterministic `risk_score` 0..100)
- endpoint-level risk gate (`PASS`/`WARN`/`BLOCK`)
- trend vs previous run for the same endpoint (risk delta + regressions)
- summary counts by status code / failure type
- optional AI summary (if enabled)

---

## Input formats

### 1) Raw cURL

```json
{ "curl": "curl ..." }
```

### 2) Normalized request

```json
{
  "method": "POST",
  "url": "https://example.com/api/login",
  "headers": {"content-type": "application/json"},
  "params": {"a": "b"},
  "body": {"username": "u", "password": "p"}
}
```

### 3) OpenAPI-driven (optional)

```json
{
  "openapi": "./openapi.json",
  "path": "/pets",
  "method": "get",
  "headers": {"accept": "application/json"},
  "path_params": {"petId": "123"},
  "query_params": {"limit": 10},
  "body": null
}
```

- OpenAPI **JSON** works by default.
- OpenAPI **YAML** requires `PyYAML` installed.
- Base URL is read from `spec.servers[0].url`.
- When using OpenAPI input, the swarm will also *optionally* validate response status codes against `operation.responses`.
- If `jsonschema` is installed (via `ai-testing-swarm[openapi]`) and the response is JSON, response bodies are validated against the OpenAPI `application/json` schema.
- If an OpenAPI requestBody schema exists for the operation, the planner can also generate **schema-based request-body fuzz cases**:
  - schema-valid edge cases (boundary values that should still validate)
  - schema-invalid variants (missing required fields, wrong types, out-of-range, etc.)
  - Controlled via CLI flags like `--no-openapi-fuzz` / `--openapi-fuzz-max-*`.
  - Override with `AI_SWARM_OPENAPI_BASE_URL` if your spec doesn’t include servers.

---

## What test cases are generated?

The swarm always includes:
- **happy_path** (baseline)

Then generates broad coverage across:
- **Method misuse**: same path with wrong HTTP methods (GET/PUT/PATCH/DELETE etc.)
- **Headers**: missing/invalid `Content-Type`, accept variations, and other header tampering
- **Auth** (if `Authorization` header exists): missing/invalid token tests
- **Body/query mutations** (per field):
  - missing / null / empty / whitespace
  - type probes (int/bool/float/array/object)
  - boundary inputs (very long strings, huge ints, negative values)
  - unicode + special character payloads
- **Security payload probes** (per field): SQLi/XSS/path traversal/log4j patterns
- **Whole-body mutations**: null body, empty object, extra unexpected field

> Output is deterministic unless OpenAI augmentation is enabled.

---

## Auth matrix runner (multiple tokens/headers)

To run the *same* request under multiple auth contexts (e.g., user/admin tokens), create `auth_matrix.yaml`:

```yaml
cases:
  - name: user
    headers:
      Authorization: "Bearer USER_TOKEN"
  - name: admin
    headers:
      Authorization: "Bearer ADMIN_TOKEN"
```

Run:

```bash
ai-test --input request.json --auth-matrix auth_matrix.yaml
```

Each auth case is written as a separate report using a `run_label` suffix (e.g. `__auth-user`).

## Safety mode (recommended for CI/demos)

Mutation testing can be noisy and may accidentally stress a real environment.
To force safe demo runs only against public test hosts:

```bash
ai-test --input request.json --public-only
```

Or via env:

```bash
export AI_SWARM_PUBLIC_ONLY=1
```

Allowed hosts in public-only mode:
- `httpbin.org`
- `postman-echo.com`
- `reqres.in`

---

## Performance features

### Parallel execution
- Enabled by default via thread pool.
- Control with:
  - `AI_SWARM_WORKERS` (default: `5`)

### Retry + backoff (flaky endpoints)
- Retries on transient errors and status codes (408/429/5xx etc.)
- Control with:
  - `AI_SWARM_RETRY_COUNT` (default: `1`)
  - `AI_SWARM_RETRY_BACKOFF_MS` (default: `250`)

### Throttling (RPS)
- Global throttle to avoid hammering a target:
  - `AI_SWARM_RPS` (default: `0` = disabled)

### Max test cap
- Avoids accidental DoS / CI timeouts:
  - `AI_SWARM_MAX_TESTS` (default: `80`)

---

## Reporting

Reports include:
- `summary.counts_by_failure_type`
- `summary.counts_by_status_code`
- `summary.slow_tests` (based on SLA)
- `meta.endpoint_risk_score` + `meta.gate_status`
- `trend.*` (previous comparison if a prior report exists)

A static dashboard index is generated at:
- `./ai_swarm_reports/index.html` (latest JSON report per endpoint, sorted by regressions/risk)

### Regression gate (CI)

To fail CI when the run regresses vs the previous JSON report for the same endpoint:

```bash
ai-test --input request.json --fail-on-regression
```

This checks `report.trend.regression_count` and exits with a non-zero code if regressions are detected.

SLA threshold:
- `AI_SWARM_SLA_MS` (default: `2000`)

Security:
- Sensitive headers are redacted in the report (Authorization/Cookie/api tokens etc.)

---

## Optional OpenAI augmentation (advanced)

### A) Generate additional test cases (planner augmentation)
Enable:

```bash
export AI_SWARM_USE_OPENAI=1
export OPENAI_API_KEY=... 
export AI_SWARM_MAX_AI_TESTS=30
```

### B) Human-readable AI summary in report
Enable:

```bash
export AI_SWARM_USE_OPENAI=1
export AI_SWARM_AI_SUMMARY=1
export OPENAI_API_KEY=...
```

Model selection:
- `AI_SWARM_OPENAI_MODEL` (default: `gpt-4.1-mini`)

---

## CLI help

```bash
ai-test --help
```

---

## Release decisions

The swarm produces a release decision:
- `APPROVE_RELEASE`
- `APPROVE_RELEASE_WITH_RISKS`
- `REJECT_RELEASE`

The decision is derived from deterministic rules (not an LLM).

---

## License

MIT