rust-crate-pipeline 4.1.1
A comprehensive system for gathering, enriching, and analyzing metadata for Rust crates using AI-powered insights, advanced caching, machine learning predictions, and modular architecture.
Requires Python: >=3.12
Keywords: rust, crate, analysis, ai, llm, pipeline, caching, ml

# Rust Crate Pipeline v4.1.1

A comprehensive, enterprise-grade system for gathering, enriching, and analyzing metadata for Rust crates using AI-powered insights, advanced caching, machine learning predictions, and modular architecture. This pipeline provides deep analysis of Rust crates with support for multiple LLM providers, intelligent caching, ML quality predictions, and comprehensive Rust code quality assessment.

## 🚀 Quick Start

### Option 1: Install via pip (Recommended for users)

```bash
# Install the package (includes automatic setup)
pip install rust-crate-pipeline

# The package will automatically run setup for all components
# You can also run setup manually:
rust-crate-pipeline --setup

# Run with your preferred LLM provider
rust-crate-pipeline --llm-provider ollama --llm-model tinyllama --crates serde tokio
```

### Option 2: Clone and run from repository (Recommended for developers)

```bash
# Clone the repository
git clone https://github.com/Superuser666-Sigil/SigilDERG-Data_Production.git
cd SigilDERG-Data_Production

# Create virtual environment (recommended)
python -m venv .venv
source .venv/bin/activate  # On Windows: .venv\Scripts\activate

# Run comprehensive setup script (installs all dependencies and configures everything)
python setup_project.py

# Or install manually:
pip install -r requirements.txt
pip install -r requirements-dev.txt
pip install -e .
python -m rust_crate_pipeline.setup_manager

# Run the pipeline (run_with_llm.py now forwards to this command)
python -m rust_crate_pipeline --llm-provider litellm --llm-model llama-3.1-8b-instruct \
  --llm-api-base http://localhost:1234/v1 --llm-api-key dummy \
  --crates-file data/crate_list.txt --limit 10
```

> ℹ️ Installing in editable mode (`pip install -e .`) ensures the
> `rust_crate_pipeline` package is discoverable without manual `sys.path`
> tweaks and enables commands like `python -m rust_crate_pipeline.main` to
> work from any directory.

## ✨ Key Features

### 🤖 **AI & Machine Learning**
- **Multi-Provider LLM Support**: Azure OpenAI, OpenAI, Anthropic, Ollama, LM Studio, Lambda.AI, and 100+ LiteLLM providers
- **ML Quality Predictor**: Automated quality scoring, security risk assessment, maintenance predictions (integrated into trust decisions)
- **Intelligent Analysis**: AI-powered insights and recommendations
- **AST-Based Code Analysis**: Tree-sitter powered Rust code analysis for accurate detection of async, unsafe, generics, and macros

### 🚀 **Performance & Caching**
- **HTTP Response Caching**: Intelligent caching of API responses via `requests-cache`
- **Cache Hit Optimization**: Faster response times for cached API calls
- **Asynchronous Processing**: Non-blocking file and network operations for improved throughput
- **Batch Processing**: Efficient batch operations for multiple crates

### 🌐 **Web Scraping & Analysis**
- **Advanced Web Scraping**: Crawl4AI + Playwright for intelligent content extraction
- **Enhanced Rust Analysis**: cargo-geiger, cargo-outdated, cargo-license, cargo-tarpaulin, cargo-deny
- **Comprehensive Tooling**: Full Rust ecosystem analysis and quality assessment

### 🔒 **Security & Trust**
- **Sigil Protocol Support**: Sacred Chain analysis with IRL trust scoring
- **Security Analysis**: Privacy and security scanning with Presidio
- **Trust Verification**: Canon registry and reputation system
- **Audit Logging**: Comprehensive audit trails for compliance

### 🏗️ **Architecture & Scalability**
- **Modular Design**: Clean separation of concerns with well-defined interfaces
- **Event-Driven**: Asynchronous processing throughout the pipeline
- **Horizontal Scaling**: Support for concurrent processing of multiple crates
- **HTTP Caching**: Intelligent HTTP response caching via `requests-cache` for improved performance
- **Modular Monolith**: Designed as a modular monolith that can be decomposed into microservices when needed

### 📊 **Monitoring & Observability**
- **Real-time Progress Tracking**: Comprehensive monitoring and error recovery
- **Structured Logging**: Detailed logging with correlation IDs
- **Audit Trails**: Complete audit logging for compliance and traceability

### 🐳 **Deployment & Operations**
- **Docker Support**: Containerized deployment with docker-compose
- **Auto-Resume Capability**: Automatically skips already processed crates
- **Batch Processing**: Configurable memory optimization and cost control
- **Production Ready**: Enterprise-grade reliability and performance

## 📋 Requirements

- **Python 3.12+** (required)
- **Git** (for repository operations)
- **Cargo** (for Rust crate analysis)
- **Playwright browsers** (auto-installed via setup)
- **Rust analysis tools** (auto-installed via setup)

### Optional Dependencies
- **Redis**: For distributed caching (recommended for production)
- **Prometheus**: For metrics collection

## 🔧 Installation & Setup

### For End Users (pip install)

The package includes automatic setup for all components:

```bash
# Install the package (includes all dependencies and automatic setup)
pip install rust-crate-pipeline

# Check setup status
rust-crate-pipeline --setup-check

# Run setup manually if needed
rust-crate-pipeline --setup --verbose-setup
```

### For Developers (repository clone)

```bash
# Clone the repository
git clone https://github.com/Superuser666-Sigil/SigilDERG-Data_Production.git
cd SigilDERG-Data_Production

# Create virtual environment (recommended)
python -m venv .venv
source .venv/bin/activate  # On Windows: .venv\Scripts\activate

# Run comprehensive setup script (recommended - installs everything automatically)
python setup_project.py

# Or install manually step-by-step:
pip install -r requirements.txt
pip install -r requirements-dev.txt
pip install -e .
python -m rust_crate_pipeline.setup_manager

# Set up environment variables (optional but recommended)
export AZURE_OPENAI_ENDPOINT="your_endpoint"
export AZURE_OPENAI_API_KEY="your_api_key"
export GITHUB_TOKEN="your_github_token"
```

> ℹ️ After running `pip install -e .`, you can invoke the CLI with
> `python -m rust_crate_pipeline.main` or the console script generated by
> the editable install without modifying `PYTHONPATH`.

## 🎯 Usage Examples

### Basic Usage with Integrated Components

```python
from rust_crate_pipeline.config import PipelineConfig
from rust_crate_pipeline.unified_pipeline import UnifiedPipeline

config = PipelineConfig(output_path="./output", batch_size=3)
pipeline_kwargs = {
    "output_dir": "./output",
    "crate_list": ["serde", "tokio", "actix-web"],
    "crate_list_path": None,
    "limit": None,
    "skip_ai": False,
    "skip_source": False,
}

pipeline = UnifiedPipeline(config, pipeline_kwargs)
await pipeline.run()
print("Artifacts written to ./output")
```

## 📚 Dataset Schema

The training dataset produced by the pipeline follows a structured JSON schema
with explicit metadata, code snippets, and analysis fields. The full schema and
example can be found in [docs/DATASET_SCHEMA.md](docs/DATASET_SCHEMA.md).

### Advanced Caching Usage

```python
from rust_crate_pipeline.utils.advanced_cache import get_cache

# Get cache instance
cache = get_cache()

# Store data with TTL and tags
await cache.set(
    "crate:serde", 
    crate_data, 
    ttl=3600,  # 1 hour
    tags=["rust", "serialization"]
)

# Retrieve data
cached_data = await cache.get("crate:serde")

# Invalidate by tags
await cache.invalidate_by_tags(["rust"])
```

### ML Quality Predictions

```python
from rust_crate_pipeline.ml.quality_predictor import get_predictor

# Get predictor instance
predictor = get_predictor()

# Predict quality metrics
prediction = predictor.predict_quality(crate_data)

print(f"Quality Score: {prediction.quality_score}")
print(f"Security Risk: {prediction.security_risk}")
print(f"Maintenance Score: {prediction.maintenance_score}")
```

### API Gateway for Microservices

```python
from rust_crate_pipeline.services.api_gateway import APIGateway

# Load configuration
with open("configs/gateway_config.json", "r") as f:
    config = json.load(f)

# Create gateway
gateway = APIGateway(config)

# Start gateway (in production)
# python rust_crate_pipeline/services/api_gateway.py --config configs/gateway_config.json
```

### Command Line Usage

```bash
# Basic analysis with caching and ML
rust-crate-pipeline --llm-provider ollama --llm-model tinyllama --crates serde tokio

# Advanced analysis with all features
rust-crate-pipeline --llm-provider azure --llm-model gpt-4o --crates actix-web

# Batch processing with auto-resume (run_with_llm.py aliases this command)
python -m rust_crate_pipeline --crates-file data/crate_list.txt --auto-resume --batch-size 5

# Force restart processing
python -m rust_crate_pipeline --crates-file data/crate_list.txt --force-restart
```

## 🔍 Enhanced Rust Analysis

The pipeline includes comprehensive Rust analysis tools:

- **cargo-geiger**: Unsafe code detection and safety scoring
- **cargo-outdated**: Dependency update recommendations
- **cargo-license**: License analysis and compliance
- **cargo-tarpaulin**: Code coverage analysis
- **cargo-deny**: Comprehensive dependency checking
- **cargo-audit**: Security vulnerability scanning
- **cargo-tree**: Dependency visualization

### Analysis Output with ML Predictions

Each crate analysis includes:

```json
{
  "enhanced_analysis": {
    "build": { "returncode": 0, "stdout": "...", "stderr": "..." },
    "test": { "returncode": 0, "stdout": "...", "stderr": "..." },
    "clippy": { "returncode": 0, "stdout": "...", "stderr": "..." },
    "geiger": { "returncode": 0, "stdout": "...", "stderr": "..." },
    "ml_predictions": {
      "quality_score": 0.85,
      "security_risk": "low",
      "maintenance_score": 0.92,
      "popularity_trend": "growing",
      "dependency_health": 0.88,
      "confidence": 0.95,
      "model_version": "1.0.0"
    },
    "insights": {
      "overall_quality_score": 0.85,
      "security_risk_level": "low",
      "code_quality": "excellent",
      "recommendations": [
        "Consider updating dependencies",
        "Review 2 unsafe code items detected by cargo-geiger"
      ]
    }
  }
}
```

## 🤖 LLM Provider Support

### Supported Providers

| Provider | Setup | Usage |
|----------|-------|-------|
| **Ollama** | `ollama serve` + `ollama pull model` | `--llm-provider ollama --llm-model tinyllama` |
| **Azure OpenAI** | Set env vars | `--llm-provider azure --llm-model gpt-4o` |
| **OpenAI** | Set `OPENAI_API_KEY` | `--llm-provider openai --llm-model gpt-4` |
| **Anthropic** | Set `ANTHROPIC_API_KEY` | `--llm-provider anthropic --llm-model claude-3` |
| **LM Studio** | Start LM Studio server | `--llm-provider litellm --llm-api-base http://localhost:1234/v1 --llm-model llama-3.1-8b-instruct` |
| **llama-cpp** | Download .gguf file | `--llm-provider llama-cpp --llm-model path/to/model.gguf` |
| **Lambda.AI** | Set `LAMBDA_API_KEY` | `--llm-provider lambda --llm-model qwen25-coder-32b` |

### Provider Configuration

```bash
# Ollama (recommended for local development)
rust-crate-pipeline --llm-provider ollama --llm-model tinyllama

# Azure OpenAI (recommended for production)
rust-crate-pipeline --llm-provider azure --llm-model gpt-4o

# OpenAI
rust-crate-pipeline --llm-provider openai --llm-model gpt-4

# Local llama-cpp model
rust-crate-pipeline --llm-provider llama-cpp --llm-model ~/models/deepseek.gguf
```

## 📊 Output and Results

### Analysis Reports & Teaching Bundles

The pipeline generates comprehensive analysis reports and optional teaching bundles per crate:

- **Basic Metadata**: Crate information, dependencies, downloads
- **Web Scraping Results**: Documentation from crates.io, docs.rs, lib.rs
- **Enhanced Analysis**: Rust tool outputs and quality metrics
- **LLM Enrichment**: AI-generated insights and recommendations
- **ML Predictions**: Quality scores, security risks, maintenance metrics
- **Sacred Chain Analysis**: Trust scoring and security assessment
- **Cache Performance**: Hit rates and optimization metrics

### Output Structure

```
output/
├── serde_analysis_report.json      # Complete analysis with ML predictions
├── tokio_analysis_report.json      # Complete analysis with ML predictions
├── checkpoint_batch_1_20250821.jsonl  # Progress checkpoints
├── pipeline_status.json            # Overall status
├── cache_metrics.json              # Cache performance metrics
└── ml_predictions_summary.json     # ML prediction summary
```

Teaching bundles structure:

```
teaching_bundles/
├── <crate_name>/
│   ├── Cargo.toml                  # Uses real crate versions
│   ├── src/lib.rs                  # Sanitized, formatted examples
│   ├── tests/basic.rs              # Auto-generated tests per topic
│   ├── README.md                   # Includes license attribution
│   ├── quality_labels.json         # Includes build/test results
│   ├── validate.sh                 # Validates compile/tests and license presence
│   └── LICENSE | COPYING           # Copied from upstream if available
└── ...
```

### Audit Logs

Comprehensive audit logs are stored in `audits/records/` for compliance and traceability.

## 🏗️ Architecture

### Modular Monolith Design

The system is designed as a modular monolith with clean separation of concerns:

```
┌─────────────────────────────────────────────────────────────┐
│                    Rust Crate Pipeline                      │
├─────────────────────────────────────────────────────────────┤
│  ┌─────────────┐ ┌─────────────┐ ┌─────────────┐           │
│  │   Core      │ │   LLM       │ │  Analysis   │           │
│  │  Pipeline   │ │  Processing │ │   Engine    │           │
│  └─────────────┘ └─────────────┘ └─────────────┘           │
│  ┌─────────────┐ ┌─────────────┐ ┌─────────────┐           │
│  │   Web       │ │   Cache     │ │   ML        │           │
│  │  Scraping   │ │   System    │ │  Predictor  │           │
│  └─────────────┘ └─────────────┘ └─────────────┘           │
│  ┌─────────────┐ ┌─────────────┐ ┌─────────────┐           │
│  │   Sigil     │ │   Audit     │ │   Utils     │           │
│  │  Protocol   │ │   System    │ │   & Tools   │           │
│  └─────────────┘ └─────────────┘ └─────────────┘           │
└─────────────────────────────────────────────────────────────┘
```

### Future: Microservices Architecture

> **Note**: API Gateway components are available (`rust_crate_pipeline/services/api_gateway.py`) but not yet integrated into the main pipeline. This is planned for future releases.

When deployed as microservices:

```
┌─────────────────────────────────────────────────────────────┐
│                    API Gateway                              │
│  ┌─────────────┐ ┌─────────────┐ ┌─────────────┐           │
│  │   Auth      │ │   Rate      │ │   Load      │           │
│  │   Service   │ │   Limiting  │ │  Balancing  │           │
│  └─────────────┘ └─────────────┘ └─────────────┘           │
└─────────────────────────────────────────────────────────────┘
                                │
        ┌───────────────────────┼───────────────────────┐
        │                       │                       │
┌───────▼────────┐    ┌─────────▼────────┐    ┌────────▼────────┐
│   Pipeline     │    │   Analysis       │    │   Scraping      │
│   Service      │    │   Service        │    │   Service       │
└────────────────┘    └──────────────────┘    └─────────────────┘
        │                       │                       │
        └───────────────────────┼───────────────────────┘
                                │
┌─────────────────────────────────────────────────────────────┐
│                    Shared Services                          │
│  ┌─────────────┐ ┌─────────────┐ ┌─────────────┐           │
│  │   Cache     │ │   Database  │ │   Message   │           │
│  │   Service   │ │   Service   │ │   Queue     │           │
│  └─────────────┘ └─────────────┘ └─────────────┘           │
└─────────────────────────────────────────────────────────────┘
```

## 🔧 Setup and Configuration

### Automatic Setup

The package includes automatic setup for all dependencies:

```bash
# Run setup (automatically runs on pip install)
rust-crate-pipeline --setup

# Check setup status
rust-crate-pipeline --setup-check

# Verbose setup with detailed output
rust-crate-pipeline --setup --verbose-setup
```

### Manual Setup

If automatic setup fails, you can run components manually:

```bash
# Install Playwright browsers
playwright install

# Install Rust analysis tools
cargo install cargo-geiger cargo-outdated cargo-license cargo-tarpaulin cargo-deny cargo-audit

# Configure environment variables
cp ~/.rust_crate_pipeline/.env.template .env
# Edit .env with your API keys
```

### Configuration Files

Setup creates configuration files in `~/.rust_crate_pipeline/`:

- `crawl4ai_config.json`: Crawl4AI settings
- `rust_tools_config.json`: Rust tool status
- `llm_providers_config.json`: LLM provider configurations
- `cache_config.json`: Cache settings and performance
- `ml_config.json`: ML model configurations
- `system_checks.json`: System compatibility results
- `.env.template`: Environment variable template

## 🐳 Docker Support

### Quick Docker Setup

```bash
# Build and run with Docker Compose
docker-compose up -d

# Run pipeline in container
docker-compose exec rust-pipeline rust-crate-pipeline --crates serde tokio
```

### Custom Docker Configuration

```dockerfile
# Use the provided Dockerfile
FROM python:3.12-slim

# Install dependencies
COPY requirements.txt .
RUN pip install -r requirements.txt

# Install Rust and tools
RUN curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
RUN cargo install cargo-geiger cargo-outdated cargo-license cargo-tarpaulin cargo-deny cargo-audit

# Install Playwright
RUN playwright install

# Copy application
COPY . /app
WORKDIR /app

# Run setup
RUN python -m rust_crate_pipeline --setup
```

## 🚀 Performance and Optimization

### Caching Performance

- **Cache Hit**: 10-100x faster response times
- **Memory Cache**: Sub-millisecond access
- **Disk Cache**: Persistent storage with intelligent eviction
- **Redis Cache**: Distributed caching for multi-instance deployments

### Batch Processing

```bash
# Optimize for memory usage
rust-crate-pipeline --batch-size 2 --max-workers 2

# Optimize for speed
rust-crate-pipeline --batch-size 10 --max-workers 8
```

### Cost Control

```bash
# Skip expensive operations
rust-crate-pipeline --skip-ai --skip-source-analysis

# Limit processing
rust-crate-pipeline --limit 50 --batch-size 5
```

## 🔍 Troubleshooting

### Common Issues

1. **Playwright browsers not installed**
   ```bash
   playwright install
   ```

2. **Rust tools not available**
   ```bash
   rust-crate-pipeline --setup
   ```

3. **LLM connection issues**
   ```bash
   # Check Ollama
   curl http://localhost:11434/api/tags
   
   # Check Azure OpenAI
   curl -H "api-key: $AZURE_OPENAI_API_KEY" "$AZURE_OPENAI_ENDPOINT/openai/deployments"
   ```

4. **Cache issues**
   ```bash
   # Clear cache
   rm -rf ~/.rust_crate_pipeline/cache/
   ```

5. **ML model issues**
   ```bash
   # ML models are automatically trained and updated during pipeline execution
   ```

### Logs and Debugging

```bash
# Enable debug logging
rust-crate-pipeline --log-level DEBUG --crates serde

# Check setup logs
cat ~/.rust_crate_pipeline/setup_results.json

# Check cache logs
cat ~/.rust_crate_pipeline/cache_metrics.json
```

## 📈 Monitoring and Metrics

### Prometheus Metrics

The system exposes comprehensive metrics:

- **Request counters**: Total requests, success/failure rates
- **Response times**: Latency histograms and percentiles
- **Cache metrics**: Hit rates, miss rates, eviction rates
- **ML metrics**: Prediction accuracy, model performance
- **System metrics**: CPU, memory, disk usage

### Health Checks

```bash
# Check overall health
curl http://localhost:8080/health

# Check specific services
curl http://localhost:8080/health/pipeline
curl http://localhost:8080/health/analysis
curl http://localhost:8080/health/scraping
```

## 🤝 Contributing

1. Fork the repository
2. Create a feature branch
3. Make your changes
4. Add tests
5. Run the test suite
6. Submit a pull request

### Development Setup

```bash
# Clone and setup development environment
git clone https://github.com/Superuser666-Sigil/SigilDERG-Data_Production.git
cd SigilDERG-Data_Production

# Install development dependencies
pip install -r requirements-dev.txt

# Run tests
pytest tests/

# Run integration tests
pytest tests/test_integration.py -v

# Run linting
black rust_crate_pipeline/
flake8 rust_crate_pipeline/
```

### Release & Packaging Checklist

Before creating wheels or source distributions make sure the trained ML artifacts
are present and up to date:

1. Regenerate the models: `python fix_ml_models.py`
2. Verify the artifact hash check: `pytest tests/test_ml_artifacts.py`
3. Build the package: `python -m build`

The automated release script (`scripts/create_release.py`) and the CI
workflows run the training step before building artifacts, but the commands
should also be executed locally when preparing manual releases.

## 📚 Documentation

- **[LLM Providers Guide](docs/README_LLM_PROVIDERS.md)**: LLM provider configuration
- **[ML Predictor Documentation](docs/ML_PREDICTOR.md)**: ML quality predictor details and synthetic data usage
- **[Architecture Guide](docs/ARCHITECTURE.md)**: Detailed architecture documentation
- **[Dataset Schema](docs/DATASET_SCHEMA.md)**: Data structure documentation
- **[Recovery Strategy](docs/RECOVERY_STRATEGY.md)**: Error recovery and resilience
- **[Remaining TODOs](docs/REMAINING_TODOS.md)**: Future enhancements and improvements
- **[Integration Examples](examples/integration_example.py)**: Usage examples

## 📄 License

This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.

## 🙏 Acknowledgments

- **Crawl4AI** for advanced web scraping capabilities
- **Playwright** for browser automation
- **Rust community** for the excellent analysis tools
- **Ollama** for local LLM serving
- **All LLM providers** for their APIs and models
- **Redis** for distributed caching
- **Prometheus** for metrics collection

## 📞 Support

- **Issues**: [GitHub Issues](https://github.com/Superuser666-Sigil/SigilDERG-Data_Production/issues)
- **Documentation**: [Wiki](https://github.com/Superuser666-Sigil/SigilDERG-Data_Production/wiki)
- **Discussions**: [GitHub Discussions](https://github.com/Superuser666-Sigil/SigilDERG-Data_Production/discussions)

---

**Rust Crate Pipeline v4.1.1** - Enterprise-grade Rust crate analysis with AI-powered insights, advanced caching, ML predictions, and microservices architecture.

**🚀 Ready for production deployment and scaling!**