mankinds-sdk 1.0.1
Python SDK for Mankinds API
License: MIT
Requires Python: >=3.8
Keywords: mankinds, api, sdk

<div align="center">


<h1>
    <img src="https://app.mankinds.io/logo-short.png" alt="Mankinds" width="32" height="32" align="top" />
    Mankinds SDK
</h1>

[![PyPI version](https://img.shields.io/pypi/v/mankinds-sdk?logo=pypi&logoColor=white)](https://pypi.org/project/mankinds-sdk/)
[![Build Status](https://img.shields.io/gitlab/pipeline/mankinds1/mankinds-sdk?branch=main&logo=gitlab&logoColor=white)](https://gitlab.com/mankinds1/mankinds-sdk/-/pipelines)
[![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg?logo=opensourceinitiative&logoColor=white)](../../LICENSE)
[![Documentation Status](https://img.shields.io/badge/docs-latest-brightgreen.svg?logo=readthedocs&logoColor=white)](https://docs.mankinds.io)

*Evaluate AI system with automated tests.*

Register an AI system, optionally attach connectors (logs, databases), import or generate your golden dataset, and run automated evaluations covering privacy, security, performance, fairness, explainability, transparency and accountability.

</div>

---

## Features

- **System Management** — Create, update, and configure AI systems with custom API endpoints
- **Endpoint Configuration** — Support for REST, SSE streaming, and multi-turn conversations
- **Dataset Generation** — Auto-generate or provide custom test scenarios
- **Evaluation** — Run evaluations with real-time polling and configurable profiles
- **Connectors** — Attach data sources (log files, Datadog, SQLite, PostgreSQL)
- **Error Handling** — Typed exceptions for all error cases

## Documentation

- [Mankinds Documentation](https://docs.mankinds.io)

## Requirements

- Python ≥ 3.8

## Installation

```bash
pip install mankinds-sdk
```

## Usage

The SDK follows a simple 3-step workflow: **create a system, generate test data, run an evaluation**.

### Initialize the Client

```python
from mankinds_sdk import MankindsClient

client = MankindsClient(api_key="mk_...")
```

| Parameter | Type | Required | Default | Description |
|-----------|------|----------|---------|-------------|
| `api_key` | `str` | Yes | — | Your API key |
| `base_url` | `str` | No | `https://app.mankinds.io` | Custom API base URL |
| `timeout` | `int` | No | `120` | Request timeout in seconds |

### Create an AI System

Register your AI system by providing its name, description, and API endpoint. The endpoint defines how your AI is called during evaluation.

```python
system = client.create_system(
    name="Customer Support Bot",
    description="A chatbot that handles order inquiries and returns for an e-commerce platform.",
    endpoint={
        "url": "https://api.example.com/chat",
        "method": "POST",
        "headers": {"Authorization": "Bearer your-token"},
        "body": {"message": "{{input}}"},
        "response": {"answer": "{{output}}"}
    }
)

system_id = system["id"]
```

Use `{{input}}` in the request body and `{{output}}` in the response mapping so test inputs and expected outputs are bound during evaluation.

### Endpoint Configuration

The **endpoint** defines how the API calls your AI system during evaluation. It's a JSON object that describes your API's request/response format.

| Field | Type | Required | Description |
|-------|------|----------|-------------|
| `url` | string | Yes | API endpoint URL |
| `method` | string | Yes | HTTP method (`POST`, `GET`, etc.) |
| `body` | object | Yes | Request body with `{{input}}` placeholder |
| `response` | object | Yes | Response mapping with `{{output}}` placeholder |
| `headers` | object | No | HTTP headers |
| `streaming` | object | No | SSE streaming configuration |
| `multiturn` | object | No | Multi-turn conversation configuration |

**Placeholders:**

- `{{input}}` in `body`: replaced with test inputs during evaluation
- `{{output}}` in `response`: indicates which field contains the AI response

```python
"body": {"message": "{{input}}"},
"response": {"answer": "{{output}}"}
```

**Streaming (SSE):**

```python
endpoint = {
    "url": "https://api.example.com/chat",
    "method": "POST",
    "body": {"message": "{{input}}"},
    "response": {"answer": "{{output}}"},
    "streaming": {
        "enabled": True,
        "format": "openai",  # "openai" | "anthropic" | "custom"
        "content_path": "choices[0].delta.content"
    }
}
```

**Multi-turn conversations:**

```python
endpoint = {
    "url": "https://api.example.com/chat",
    "method": "POST",
    "body": {"message": "{{input}}", "session_id": "{{session}}"},
    "response": {"answer": "{{output}}"},
    "multiturn": {
        "type": "session_id",  # "none" | "session_id" | "history"
        "field": "conversation_id",
        "location": "body"
    }
}
```

### Generate Evaluation Dataset

Test scenarios can be auto-generated based on your system description, or you can provide custom scenarios.

**Auto-generate scenarios:**

```python
dataset = client.generate_dataset(system_id, num_scenarios=20)
```

**Provide custom scenarios:**

```python
dataset = client.generate_dataset(
    system_id,
    scenarios=[
        {"input": "Where is my order?", "outputs": ["I can help you track your order."]},
        {"input": "I want a refund", "outputs": ["I'll process your refund request."]}
    ]
)
```

**Refine an existing dataset:**

```python
dataset = client.update_dataset(
    system_id,
    orientation="Add more edge cases about payment failures"
)
```

> **Note:** `generate_dataset` requires a validated system description. If validation fails, a `DescriptionNotValidatedError` is raised with recommendations.

### Run Evaluation

Start an evaluation to test your AI system. By default, the call blocks until the evaluation completes.

**Block until complete (default):**

```python
result = client.evaluate(system_id)
print(f"Score: {result['summary']}")
```

**Start without waiting:**

```python
run_info = client.evaluate(system_id, wait=False)
run_id = run_info["run_id"]

# Check status later
result = client.get_evaluation(run_id)
print(f"Status: {result['status']}")
```

**With specific thematics:**

```python
result = client.evaluate(
    system_id,
    thematics_config={
        "explainability": {"justification": {"nb_tests": 5}},
        "robustness": {"prompt_injection": {"nb_tests": 10}}
    }
)
```

**With evaluation profile:**

```python
result = client.evaluate(system_id, profile="extended")
```

**With progress callback:**

```python
result = client.evaluate(
    system_id,
    poll_interval=10,
    on_poll=lambda status, elapsed: print(f"  {status} ({elapsed}s)")
)
```

### Connectors

**Connectors** attach external data sources (logs, databases) to your system for richer evaluation context.

**File logs:**

```python
from mankinds_sdk.connectors import FileConnector

connector = FileConnector(file_path="/path/to/logs.json")
client.add_connector(system_id, connector)
```

**Datadog logs:**

```python
from mankinds_sdk.connectors import DatadogConnector

connector = DatadogConnector(
    api_key="dd-api-key",
    app_key="dd-app-key",
    site="datadoghq.eu",  # default
)
client.add_connector(system_id, connector)
```

**SQLite database:**

```python
from mankinds_sdk.connectors import SqliteConnector

connector = SqliteConnector(file_path="/path/to/database.db")
client.add_connector(system_id, connector)
```

**PostgreSQL database:**

```python
from mankinds_sdk.connectors import PostgresqlConnector

connector = PostgresqlConnector(
    host="localhost",
    database="mydb",
    user="admin",
    password="secret",
    port=5432,
)
client.add_connector(system_id, connector)
```

**Manage connectors:**

```python
# List all connectors
connectors = client.get_connectors(system_id)

# Update a connector
connector = FileConnector(file_path="/path/to/new-logs.json")
client.update_connector(system_id, connector)

# Remove a connector
client.delete_connector(system_id, connector)
```

> Only one connector per category (logs, database) is allowed per system. Adding a duplicate raises `ConnectorAlreadyExistsError`.

## Complete Example

```python
from mankinds_sdk import MankindsClient
from mankinds_sdk.connectors import FileConnector

client = MankindsClient(api_key="mk_...")

# Create system
system = client.create_system(
    name="Support Bot",
    description="A customer support chatbot for order tracking and returns.",
    endpoint={
        "url": "https://api.example.com/chat",
        "method": "POST",
        "body": {"message": "{{input}}"},
        "response": {"answer": "{{output}}"}
    }
)
system_id = system["id"]

# Attach production logs
connector = FileConnector(file_path="./logs/production.json")
client.add_connector(system_id, connector)

# Generate dataset and evaluate
dataset = client.generate_dataset(system_id, num_scenarios=15)
result = client.evaluate(system_id, profile="extended")

print(f"Status: {result['status']}")
print(f"Score: {result['summary']}")
```

## API Reference

### MankindsClient

| Method | Description |
|--------|-------------|
| `get_system(system_id)` | Get system details and configuration |
| `create_system(name, description, endpoint)` | Create a new AI system |
| `update_system(system_id, name?, description?, endpoint?)` | Update an existing system |
| `generate_dataset(system_id, num_scenarios?, scenarios?)` | Generate and validate evaluation scenarios |
| `update_dataset(system_id, orientation?, scenarios?)` | Refine or replace dataset scenarios |
| `evaluate(system_id, ...)` | Run an evaluation |
| `get_evaluation(run_id)` | Get evaluation status and results |
| `add_connector(system_id, connector)` | Add a data source connector |
| `get_connectors(system_id)` | List all connectors for a system |
| `update_connector(system_id, connector)` | Update a connector |
| `delete_connector(system_id, connector)` | Remove a connector |

### Exceptions

| Exception | When Raised |
|-----------|-------------|
| `CredentialsError` | Missing API key |
| `AuthenticationError` | Invalid or expired API key (401) |
| `NotFoundError` | Resource not found (404) |
| `ValidationError` | Request validation failed (422) |
| `RateLimitError` | Too many requests (429) |
| `ServerError` | Server error (5xx) |
| `InvalidEndpointError` | Endpoint missing required fields |
| `EndpointNotConfiguredError` | Evaluation without endpoint |
| `DescriptionNotValidatedError` | Dataset generation before validation |
| `ConnectorAlreadyExistsError` | Duplicate connector category |

```python
from mankinds_sdk.exceptions import (
    CredentialsError,
    AuthenticationError,
    InvalidEndpointError,
    DescriptionNotValidatedError,
    ConnectorAlreadyExistsError,
)

try:
    result = client.evaluate(system_id)
except AuthenticationError:
    print("Invalid API key")
except InvalidEndpointError as e:
    print(f"Missing fields: {e.missing_fields}")
except DescriptionNotValidatedError as e:
    print(f"Fix description: {e.recommendations}")
```

---

<div align="center">

## License

[MIT](../../LICENSE)

© 2026 [Mankinds](https://mankinds.io). All rights reserved.

</div>