codibox 1.3.0
A reusable package for executing Python code in Docker containers and extracting charts/images
License: MIT
Requires Python: >=3.10
Keywords: docker, code-execution, sandbox, jupyter, notebook, matplotlib, charts, images, python-execution

# Codibox - Safe Code Execution Package

[![Python 3.10+](https://img.shields.io/badge/python-3.10+-blue.svg)](https://www.python.org/downloads/)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
[![CI](https://github.com/otmane-elbourki/codibox/workflows/CI/badge.svg)](https://github.com/otmane-elbourki/codibox/actions)
[![Ruff](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/astral-sh/ruff/main/assets/badge/v2.json)](https://github.com/astral-sh/ruff)

A reusable Python package for executing Python code in Docker containers or on the host machine, with automatic chart/image extraction and result processing.

## 📸 Screenshots

### Streamlit Chart Generator Example

<div align="center">
  <img src="assets/msedge_t8P7Uoe4tf.png" alt="Streamlit Chart Generator" width="48%" />
  <img src="assets/msedge_ZhiiatsgMj.png" alt="Streamlit Chart Generator Results" width="48%" />
</div>

The [Streamlit Chart Generator example](examples/ai_analyst/streamlit_chart_generator.py) demonstrates an interactive dashboard for generating charts from data using codibox.

## 🎯 Features

- **Dual Backend Support**: Choose between Docker (secure, isolated) or Host (fast, direct execution)
- **Automatic Image Extraction**: Automatically finds and extracts charts/images from code execution
- **Markdown Processing**: Converts notebook output to markdown with embedded images
- **CSV File Extraction**: Automatically extracts generated CSV/Excel files
- **Dependency Management**: Auto-installs required packages for host mode
- **Error Handling**: Comprehensive error handling with fallbacks
- **Backward Compatible**: `DockerCodeExecutor` alias maintained for existing code

## 📦 Installation

### From PyPI (Recommended)

```bash
pip install codibox
```

### From Source

```bash
# Clone the repository
git clone https://github.com/otmane-elbourki/codibox.git
cd codibox

# Install in editable mode
pip install -e .
```

### Development Installation

```bash
pip install -e ".[dev]"
```

## 🚀 Quick Start

### Docker Backend (Default - Secure)

```python
from codibox import CodeExecutor

# Initialize executor with Docker backend
executor = CodeExecutor(backend="docker", container_name="sandbox")

# Execute code
result = executor.execute(
    code="""
    import matplotlib.pyplot as plt
    import numpy as np
    
    x = np.linspace(0, 10, 100)
    y = np.sin(x)
    
    plt.figure(figsize=(10, 6))
    plt.plot(x, y, label='sin(x)')
    plt.title('Sine Wave')
    plt.savefig('temp_code_files/chart.png')
    plt.close()
    """
)

# Check results
if result.success:
    print(f"✅ Success! Generated {len(result.images)} images")
    for img in result.images:
        print(f"  📊 {img}")
    print(f"📄 Markdown: {result.markdown[:100]}...")
else:
    print(f"❌ Failed: {result.stderr}")
```

### Host Backend (Fast - Direct Execution)

```python
from codibox import CodeExecutor

# Initialize executor with Host backend (fast mode)
executor = CodeExecutor(backend="host", auto_install_deps=True)

# Execute code (same API as Docker backend)
result = executor.execute(
    code="""
    import matplotlib.pyplot as plt
    import numpy as np
    
    x = np.linspace(0, 10, 100)
    y = np.cos(x)
    
    plt.figure(figsize=(10, 6))
    plt.plot(x, y)
    plt.title('Cosine Wave')
    plt.savefig('temp_code_files/cosine.png')
    plt.close()
    """
)

# Results work the same way
print(f"Images: {result.images}")
print(f"Success: {result.success}")
```

### Backward Compatibility

```python
# Old code still works!
from codibox import DockerCodeExecutor

executor = DockerCodeExecutor(container_name="sandbox")
result = executor.execute(code="...")
```

## 📋 API Reference

### `CodeExecutor`

Main class for executing Python code with dual backend support.

#### Initialization

```python
CodeExecutor(
    backend: str = "docker",              # "docker" or "host"
    container_name: str = "sandbox",      # Docker container name (docker only)
    host_temp_dir: str = "/tmp",          # Temporary directory on host
    image_output_dir: str = "temp_code_files",  # Directory for images
    timeout: Optional[int] = 300,        # Execution timeout (seconds)
    cleanup_on_error: bool = True,       # Cleanup on error
    auto_setup: bool = False,             # Auto-setup Docker container (docker only)
    auto_install_deps: bool = True,       # Auto-install dependencies (host only)
)
```

#### Methods

##### `execute(code, input_files=None, working_dir=None) -> ExecutionResult`

Execute Python code and return results.

**Parameters:**
- `code` (str): Python code to execute
- `input_files` (List[str], optional): Input files to copy to container/host
- `working_dir` (str, optional): Working directory (default: `/home/sandboxuser` for Docker, temp dir for Host)

**Returns:** `ExecutionResult` object

##### `check_container() -> bool`

Check if Docker container is running (Docker backend only).

##### `get_container_status() -> Dict[str, Any]`

Get detailed status of Docker container (Docker backend only).

**Returns:** Dictionary with `exists`, `running`, `status`, `image`

##### `setup_container(image_name="python_sandbox:latest", force_rebuild=False) -> bool`

Set up Docker container by building image and starting container (Docker backend only).

### `ExecutionResult`

Result object returned by `execute()` method.

```python
@dataclass
class ExecutionResult:
    # Raw outputs
    markdown: str                    # Raw markdown from notebook execution
    success: bool                    # Whether execution succeeded
    images: List[str]                # List of image file paths on host
    csv_files: List[str]             # List of generated CSV/Excel files
    stdout: Optional[str]            # Standard output
    stderr: Optional[str]            # Standard error
    execution_time: Optional[float]  # Execution time in seconds
    
    # Processed outputs
    markdown_processed: Optional[str]      # Markdown with resolved image paths
    markdown_with_images: Optional[str]    # Markdown with base64-embedded images
    image_map: Optional[Dict[str, str]]    # Mapping: markdown_ref -> actual_path
```

### `ImageProcessor`

Utility class for processing images from execution results.

```python
from codibox import ImageProcessor

processor = ImageProcessor(image_base_dir="/tmp/temp_code_files")

# Find all images
images = processor.find_images()

# Process markdown to embed images
processed_markdown = processor.process_markdown(result.markdown)

# Get HTML for all images
html = processor.get_all_images_html()
```

## 🔧 Configuration Examples

### Docker Backend Setup

```python
from codibox import CodeExecutor

# Basic setup
executor = CodeExecutor(backend="docker", container_name="sandbox")

# Auto-setup container if not running
executor = CodeExecutor(
    backend="docker",
    container_name="sandbox",
    auto_setup=True  # Automatically builds/starts container
)

# Manual container setup
executor = CodeExecutor(backend="docker")
if not executor.check_container():
    executor.setup_container()

# Custom timeout
executor = CodeExecutor(backend="docker", timeout=600)  # 10 minutes
```

### Host Backend Setup

```python
from codibox import CodeExecutor

# Basic setup (auto-installs dependencies)
executor = CodeExecutor(backend="host")

# Disable auto-installation
executor = CodeExecutor(
    backend="host",
    auto_install_deps=False  # You must install dependencies manually
)

# Custom temp directory
executor = CodeExecutor(
    backend="host",
    host_temp_dir="/custom/tmp"
)
```

## 📊 Usage Examples

### With Input Files

```python
executor = CodeExecutor(backend="docker")

result = executor.execute(
    code="""
    import pandas as pd
    import matplotlib.pyplot as plt
    
    df = pd.read_csv('data.csv')
    plt.hist(df['age'], bins=20, edgecolor='black')
    plt.title('Age Distribution')
    plt.savefig('temp_code_files/age_dist.png')
    plt.close()
    """,
    input_files=["datasets/customer_data.csv"]
)

print(f"Generated {len(result.images)} charts")
print(f"CSV files: {result.csv_files}")
```

### Processing Images

```python
from codibox import CodeExecutor, ImageProcessor

executor = CodeExecutor(backend="host")
result = executor.execute(code="...")

# Process images
processor = ImageProcessor()
markdown_with_images = processor.process_markdown(result.markdown)

# Or get all images as HTML
images_html = processor.get_all_images_html()
print(images_html)
```

### Accessing Processed Results

```python
result = executor.execute(code="...")

# Raw markdown (with image references)
print(result.markdown)

# Processed markdown (with resolved paths)
if result.markdown_processed:
    print(result.markdown_processed)

# Markdown with embedded base64 images
if result.markdown_with_images:
    print(result.markdown_with_images)

# Image mapping
if result.image_map:
    for ref, path in result.image_map.items():
        print(f"{ref} -> {path}")
```

## 🐳 Docker Backend Details

### Container Setup

The Docker backend requires a running container with required packages installed. You have three options:

#### Option 1: Manual Docker Setup (Recommended)

**Step 1: Create and start the container:**
```bash
docker run -d --name sandbox --network none --cap-drop all \
  --pids-limit 124 --tmpfs /tmp:rw,size=124M \
  python:3.10-slim sleep infinity
```

**Step 2: Install required packages:**
```bash
docker exec sandbox pip install pandas matplotlib seaborn jupyter nbconvert ipynb-py-convert
```

**Step 3: Verify container is running:**
```bash
docker ps -a | grep sandbox
# If container exists but is stopped, start it:
docker start sandbox
```

**Step 4: Use in Python:**
```python
from codibox import CodeExecutor

executor = CodeExecutor(backend="docker", container_name="sandbox")
result = executor.execute(code="...")
```

#### Option 2: Auto-setup (Development)

The package can automatically set up the container using the included Dockerfile:

```python
executor = CodeExecutor(backend="docker", auto_setup=True)
# Container will be built and started automatically if not running
```

#### Option 3: Use Existing Container

If you already have a container with the required packages:

```python
executor = CodeExecutor(backend="docker", container_name="my_container")
```

### Container Requirements

The container must have:
- **Python 3.10+**
- **Required packages:** `pandas`, `numpy`, `matplotlib`, `jupyter`, `nbconvert`, `ipynb-py-convert`
- **Working directory:** `/tmp` (used by default)

**Note:** The package includes a `Dockerfile` and `requirements.txt` in the `docker/` directory for building a compatible container, but manual setup with `python:3.10-slim` is faster and recommended.

## 💻 Host Backend Details

### Automatic Dependency Management

The Host backend automatically checks and installs required packages:
- `matplotlib`
- `pandas`
- `numpy`
- `jupyter`
- `nbconvert`
- `ipynb-py-convert`

### Python 3.13 Compatibility

The Host backend automatically handles Python 3.13 SQLite issues by falling back to direct Python execution when Jupyter fails.

### Security Considerations

⚠️ **Important**: The Host backend runs code with user permissions and has no isolation. Use Docker backend for untrusted code.

## 🔄 Backend Comparison

| Feature | Docker Backend | Host Backend |
|---------|---------------|--------------|
| **Security** | ✅ Isolated, secure | ⚠️ Runs with user permissions |
| **Speed** | ~2-5 seconds | ~0.5-2 seconds |
| **Setup** | Requires Docker | No setup needed |
| **Dependencies** | Pre-installed in container | Auto-installed on first use |
| **Isolation** | ✅ Full isolation | ❌ No isolation |
| **Network** | ❌ No network access | ✅ Full network access |
| **Use Case** | Production, untrusted code | Development, trusted code |

## 📝 Examples

See the `examples/` directory for complete usage examples:

- **`basic_usage.py`** - Simple code execution examples
- **`ai_analyst/`** - Complete Streamlit dashboard application
- **`README.md`** - Detailed examples documentation

### Running Examples

```bash
# Setup Docker container (if using Docker backend)
cd examples
./setup_example.sh

# Run basic examples
python basic_usage.py

# Run Streamlit app
cd ai_analyst
streamlit run streamlit_chart_generator.py
```

## 🧪 Testing

Run the test suite to verify both backends:

```bash
python3 test/test_both_backends.py
```

This will test:
- Docker backend execution
- Host backend execution
- Image generation and extraction
- Markdown processing
- Error handling

## 📚 Advanced Usage

### Custom Working Directory

```python
# Docker backend
result = executor.execute(
    code="...",
    working_dir="/custom/path"
)

# Host backend
executor = CodeExecutor(
    backend="host",
    host_temp_dir="/custom/tmp"
)
```

### Error Handling

```python
result = executor.execute(code="...")

if not result.success:
    print(f"Execution failed: {result.stderr}")
    print(f"Execution time: {result.execution_time}s")
else:
    print(f"Success! Generated {len(result.images)} images")
```

### Container Management

```python
executor = CodeExecutor(backend="docker")

# Check container status
status = executor.get_container_status()
print(f"Container exists: {status['exists']}")
print(f"Container running: {status['running']}")

# Setup container
if not executor.check_container():
    executor.setup_container(force_rebuild=True)
```

## 🔍 Troubleshooting

### Docker Backend Issues

**Container not found:**
```python
# Auto-setup container
executor = CodeExecutor(backend="docker", auto_setup=True)

# Or manually setup
executor.setup_container()
```

**Images not extracted:**
- Ensure code saves images to `temp_code_files/` directory
- Check container working directory matches executor configuration

### Host Backend Issues

**Dependencies not installed:**
```python
# Enable auto-installation
executor = CodeExecutor(backend="host", auto_install_deps=True)
```

**Python 3.13 SQLite errors:**
- Automatically handled with fallback to direct Python execution
- No action needed

**Images not found:**
- Ensure `temp_code_files/` directory exists (created automatically)
- Check image paths in code match `image_output_dir` setting

## 📦 Package Structure

```
.
├── __init__.py              # Package initialization
├── executor.py              # CodeExecutor class
├── image_processor.py       # Image processing utilities
├── type_definitions.py      # Type definitions
├── setup.py                 # Package setup configuration
├── pyproject.toml           # Modern Python packaging config
├── Makefile                 # Development commands
├── docker/
│   ├── Dockerfile          # Docker image definition
│   └── requirements.txt    # Docker dependencies
├── scripts/
│   └── bump_version.py     # Version management script
├── test/
│   └── test_both_backends.py  # Test script
├── examples/               # Usage examples
│   ├── basic_usage.py
│   └── ai_analyst/         # Streamlit dashboard example
└── .github/
    └── workflows/          # CI/CD workflows
        ├── ci.yml          # Continuous Integration
        └── publish.yml     # PyPI publishing
```

## 🗺️ Future Roadmap

### Performance & Scalability
- **Container Pool Executor** - Pre-warmed container pool for parallel execution (target: 1-3s per execution, 10x throughput)
- **FastAPI Microservice** - RESTful API with async execution, job queue, and WebSocket support
- **Optimizations** - Shared volumes, batch operations, result caching, parallel extraction

### Production Features
- **Monitoring** - Prometheus metrics, Grafana dashboards, health checks
- **Security** - API authentication, rate limiting, enhanced container hardening
- **Reliability** - Auto-recovery, circuit breakers, retry logic, dead letter queues
- **Deployment** - Docker Compose, Kubernetes manifests, auto-scaling

### Current Performance
- Docker: ~2-5s per execution | Host: ~0.5-2s per execution
- **Target:** <1s with pool, <500ms with warm containers
- **Throughput:** 1 req/s → 50-100 req/s (with horizontal scaling)

## 🤝 Contributing

Contributions are welcome! Please see [CONTRIBUTING.md](CONTRIBUTING.md) for guidelines.

### Development Setup

```bash
# Install development dependencies (includes pre-commit hooks)
make install-dev

# Run all checks (format, lint, type-check, test)
make ci

# Individual commands
make test        # Run tests
make format      # Format code with ruff
make lint        # Lint code with ruff
make type-check  # Type checking with mypy
make build       # Build package
```

### CI/CD

The project uses GitHub Actions for automated testing and publishing:

- **Continuous Integration**: Runs on every push/PR
  - Tests on Python 3.10, 3.11, 3.12
  - Code formatting and linting checks
  - Build verification

- **PyPI Publishing**: Automatically publishes when you create a GitHub release
  - No manual uploads needed
  - Uses trusted publishing (no API tokens required)

### Pre-commit Hooks

Pre-commit hooks automatically check code quality before each commit:
- Code formatting (ruff format)
- Linting (ruff check)
- Type checking (mypy)
- Basic file validation

Install with: `make install-dev` or `pre-commit install`

### Releasing a New Version

1. **Bump version**:
   ```bash
   make bump-patch   # 1.1.0 -> 1.1.1
   make bump-minor   # 1.1.0 -> 1.2.0
   make bump-major   # 1.1.0 -> 2.0.0
   ```

2. **Commit and tag**:
   ```bash
   git commit -am "Bump version to X.Y.Z"
   git tag -a vX.Y.Z -m "Release vX.Y.Z"
   git push && git push --tags
   ```

3. **Create GitHub Release**:
   - Go to GitHub → Releases → Draft new release
   - Select the tag (vX.Y.Z)
   - Add release notes
   - Publish (triggers automatic PyPI publish)

**Version Numbering**: Follow [Semantic Versioning](https://semver.org/)
- **MAJOR** (X.0.0): Breaking changes
- **MINOR** (0.X.0): New features, backward compatible
- **PATCH** (0.0.X): Bug fixes, backward compatible

## 📄 License

This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.

## 👤 Author

**Otmane El Bourki**
- Email: otmane.elbourki@gmail.com
- GitHub: [@otmane-elbourki](https://github.com/otmane-elbourki)

## 🔗 Links

- **PyPI**: https://pypi.org/project/codibox/
- **GitHub**: https://github.com/otmane-elbourki/codibox
- **Issues**: https://github.com/otmane-elbourki/codibox/issues