Spaces:
Running
on
CPU Upgrade
Running
on
CPU Upgrade
| # Deep Analysis API Documentation | |
| ## Overview | |
| The Deep Analysis system provides advanced multi-agent analytical capabilities that automatically generate comprehensive reports based on user goals. The system uses DSPy (Declarative Self-improving Language Programs) to orchestrate multiple AI agents and create detailed analytical insights. | |
| ## Key Features | |
| - **Multi-Agent Analysis**: Orchestrates multiple specialized agents (preprocessing, statistical analysis, machine learning, visualization) | |
| - **Template Integration**: Uses the user's active templates/agents for analysis | |
| - **Streaming Progress**: Real-time progress updates during analysis execution | |
| - **Report Persistence**: Stores complete analysis reports in database with metadata | |
| - **HTML Export**: Generates downloadable HTML reports with visualizations | |
| - **Credit Tracking**: Monitors token usage, costs, and credits consumed | |
| ## Template Integration | |
| The deep analysis system integrates with the user's active templates through the agent system: | |
| 1. **Agent Selection**: Uses agents from the user's active template preferences (configured via `/templates` endpoints) | |
| 2. **Default Agents**: Falls back to system default agents if user hasn't configured preferences: | |
| - `preprocessing` (both individual and planner variants) | |
| - `statistical_analytics` (both individual and planner variants) | |
| - `sk_learn` (both individual and planner variants) | |
| - `data_viz` (both individual and planner variants) | |
| 3. **Template Limits**: Respects the 10-template limit for planner performance optimization | |
| 4. **Dynamic Planning**: The planner automatically selects the most appropriate agents based on the analysis goal and available templates | |
| ## Analysis Flow | |
| The deep analysis process follows these steps: | |
| 1. **Question Generation** (20% progress): Generates 5 targeted analytical questions based on the user's goal | |
| 2. **Planning** (40% progress): Creates an optimized execution plan using available agents | |
| 3. **Agent Execution** (60% progress): Executes analysis using user's active templates | |
| 4. **Code Synthesis** (80% progress): Combines and optimizes code from all agents | |
| 5. **Code Execution** (85% progress): Runs the synthesized analysis code | |
| 6. **Synthesis** (90% progress): Synthesizes results into coherent insights | |
| 7. **Conclusion** (100% progress): Generates final conclusions and recommendations | |
| --- | |
| ## API Endpoints | |
| ### Create Deep Analysis Report | |
| **POST** `/deep_analysis/reports` | |
| Creates a new deep analysis report in the database. | |
| **Request Body:** | |
| ```json | |
| { | |
| "report_uuid": "string", | |
| "user_id": 123, | |
| "goal": "Analyze customer churn patterns", | |
| "status": "completed", | |
| "deep_questions": "1. What factors...\n2. How does...", | |
| "deep_plan": "{\n \"@preprocessing\": {\n \"create\": [...],\n \"use\": [...],\n \"instruction\": \"...\"\n }\n}", | |
| "summaries": ["Agent summary 1", "Agent summary 2"], | |
| "analysis_code": "import pandas as pd\n# Analysis code...", | |
| "plotly_figures": [{"data": [...], "layout": {...}}], | |
| "synthesis": ["Synthesis result 1"], | |
| "final_conclusion": "## Conclusion\nThe analysis reveals...", | |
| "html_report": "<html>...</html>", | |
| "report_summary": "Brief summary of findings", | |
| "progress_percentage": 100, | |
| "duration_seconds": 120, | |
| "credits_consumed": 5, | |
| "error_message": null, | |
| "model_provider": "anthropic", | |
| "model_name": "claude-sonnet-4-20250514", | |
| "total_tokens_used": 15000, | |
| "estimated_cost": 0.25, | |
| "steps_completed": ["questions", "planning", "execution", "synthesis", "conclusion"] | |
| } | |
| ``` | |
| **Response:** | |
| ```json | |
| { | |
| "report_id": 1, | |
| "report_uuid": "uuid-string", | |
| "user_id": 123, | |
| "goal": "Analyze customer churn patterns", | |
| "status": "completed", | |
| "start_time": "2024-01-01T12:00:00Z", | |
| "end_time": "2024-01-01T12:02:00Z", | |
| "duration_seconds": 120, | |
| "report_summary": "Brief summary of findings", | |
| "created_at": "2024-01-01T12:02:00Z", | |
| "updated_at": "2024-01-01T12:02:00Z" | |
| } | |
| ``` | |
| ### Get Deep Analysis Reports | |
| **GET** `/deep_analysis/reports` | |
| Retrieves a list of deep analysis reports with optional filtering. | |
| **Query Parameters:** | |
| - `user_id` (optional): Filter by user ID | |
| - `limit` (optional): Number of reports to return (1-100, default: 10) | |
| - `offset` (optional): Number of reports to skip (default: 0) | |
| - `status` (optional): Filter by status ("pending", "running", "completed", "failed") | |
| **Response:** | |
| ```json | |
| [ | |
| { | |
| "report_id": 1, | |
| "report_uuid": "uuid-string", | |
| "user_id": 123, | |
| "goal": "Analyze customer churn patterns", | |
| "status": "completed", | |
| "start_time": "2024-01-01T12:00:00Z", | |
| "end_time": "2024-01-01T12:02:00Z", | |
| "duration_seconds": 120, | |
| "report_summary": "Brief summary of findings", | |
| "created_at": "2024-01-01T12:02:00Z", | |
| "updated_at": "2024-01-01T12:02:00Z" | |
| } | |
| ] | |
| ``` | |
| ### Get User Historical Reports | |
| **GET** `/deep_analysis/reports/user_historical` | |
| Retrieves all historical deep analysis reports for a specific user. | |
| **Query Parameters:** | |
| - `user_id`: User ID (required) | |
| - `limit` (optional): Number of reports to return (1-100, default: 50) | |
| ### Get Report by ID | |
| **GET** `/deep_analysis/reports/{report_id}` | |
| Retrieves a complete deep analysis report by ID. | |
| **Query Parameters:** | |
| - `user_id` (optional): Ensures report belongs to specified user | |
| **Response:** | |
| ```json | |
| { | |
| "report_id": 1, | |
| "report_uuid": "uuid-string", | |
| "user_id": 123, | |
| "goal": "Analyze customer churn patterns", | |
| "status": "completed", | |
| "start_time": "2024-01-01T12:00:00Z", | |
| "end_time": "2024-01-01T12:02:00Z", | |
| "duration_seconds": 120, | |
| "deep_questions": "1. What factors contribute to churn?\n2. How does churn vary by segment?", | |
| "deep_plan": "{\n \"@preprocessing\": {...},\n \"@statistical_analytics\": {...}\n}", | |
| "summaries": ["Agent performed data cleaning...", "Statistical analysis revealed..."], | |
| "analysis_code": "import pandas as pd\n# Complete analysis code", | |
| "plotly_figures": [{"data": [...], "layout": {...}}], | |
| "synthesis": ["The analysis shows clear patterns..."], | |
| "final_conclusion": "## Conclusion\nCustomer churn is primarily driven by...", | |
| "html_report": "<html>...</html>", | |
| "report_summary": "Analysis of customer churn patterns reveals...", | |
| "progress_percentage": 100, | |
| "credits_consumed": 5, | |
| "error_message": null, | |
| "model_provider": "anthropic", | |
| "model_name": "claude-sonnet-4-20250514", | |
| "total_tokens_used": 15000, | |
| "estimated_cost": 0.25, | |
| "steps_completed": ["questions", "planning", "execution", "synthesis", "conclusion"], | |
| "created_at": "2024-01-01T12:02:00Z", | |
| "updated_at": "2024-01-01T12:02:00Z" | |
| } | |
| ``` | |
| ### Get Report by UUID | |
| **GET** `/deep_analysis/reports/uuid/{report_uuid}` | |
| Retrieves a complete deep analysis report by UUID. Same response format as get by ID. | |
| ### Delete Report | |
| **DELETE** `/deep_analysis/reports/{report_id}` | |
| Deletes a deep analysis report. | |
| **Query Parameters:** | |
| - `user_id` (optional): Ensures report belongs to specified user | |
| **Response:** | |
| ```json | |
| { | |
| "message": "Report 1 deleted successfully" | |
| } | |
| ``` | |
| ### Update Report Status | |
| **PUT** `/deep_analysis/reports/{report_id}/status` | |
| Updates the status of a deep analysis report. | |
| **Request Body:** | |
| ```json | |
| { | |
| "status": "completed" | |
| } | |
| ``` | |
| **Valid Status Values:** | |
| - `pending`: Analysis queued but not started | |
| - `running`: Analysis in progress | |
| - `completed`: Analysis finished successfully | |
| - `failed`: Analysis encountered errors | |
| ### Get HTML Report | |
| **GET** `/deep_analysis/reports/uuid/{report_uuid}/html` | |
| Retrieves only the HTML report content for a specific analysis. | |
| **Query Parameters:** | |
| - `user_id` (optional): Ensures report belongs to specified user | |
| **Response:** | |
| ```json | |
| { | |
| "html_report": "<html>...</html>", | |
| "filename": "deep_analysis_report_20240101_120200.html" | |
| } | |
| ``` | |
| ### Download HTML Report | |
| **POST** `/deep_analysis/download_from_db/{report_uuid}` | |
| Downloads the HTML report as a file attachment. | |
| **Query Parameters:** | |
| - `user_id` (optional): Ensures report belongs to specified user | |
| **Response:** | |
| - Content-Type: `text/html; charset=utf-8` | |
| - Content-Disposition: `attachment; filename="deep_analysis_report_TIMESTAMP.html"` | |
| --- | |
| ## Deep Analysis Module Architecture | |
| ### DSPy Signatures | |
| The system uses several DSPy signatures for different analysis phases: | |
| #### 1. `deep_questions` | |
| Generates 5 targeted analytical questions based on the user's goal and dataset structure. | |
| #### 2. `deep_planner` | |
| Creates an optimized execution plan using the user's active templates/agents. The planner: | |
| - Verifies feasibility using available datasets and agent descriptions | |
| - Batches similar questions per agent call for efficiency | |
| - Reuses outputs across questions to minimize agent calls | |
| - Defines clear variable flow and dependencies between agents | |
| #### 3. `deep_code_synthesizer` | |
| Combines and optimizes code from multiple agents: | |
| - Fixes errors and inconsistencies between agent outputs | |
| - Ensures proper data flow and type handling | |
| - Converts all visualizations to Plotly format | |
| - Adds comprehensive error handling and validation | |
| #### 4. `deep_synthesizer` | |
| Synthesizes analysis results into coherent insights and findings. | |
| #### 5. `final_conclusion` | |
| Generates final conclusions and strategic recommendations based on all analysis results. | |
| ### Streaming Analysis | |
| The `execute_deep_analysis_streaming` method provides real-time progress updates: | |
| ```python | |
| async for update in deep_analysis.execute_deep_analysis_streaming(goal, dataset_info, session_df): | |
| if update["step"] == "questions": | |
| # Handle questions generation progress | |
| elif update["step"] == "planning": | |
| # Handle planning progress | |
| elif update["step"] == "agent_execution": | |
| # Handle agent execution progress | |
| # ... handle other steps | |
| ``` | |
| ### Integration with User Templates | |
| The deep analysis system integrates with user templates in several ways: | |
| 1. **Agent Discovery**: Retrieves user's active template preferences from the database | |
| 2. **Dynamic Planning**: The planner uses available agents to create optimal execution plans | |
| 3. **Template Validation**: Ensures all referenced agents exist in the user's active templates | |
| 4. **Fallback Handling**: Uses default agents if user preferences are incomplete | |
| 5. **Performance Optimization**: Respects template limits for efficient execution | |
| ### Error Handling | |
| The system includes comprehensive error handling: | |
| - **Code Execution Errors**: Automatically attempts to fix and retry failed code | |
| - **Template Missing**: Falls back to default agents if user templates are unavailable | |
| - **Timeout Protection**: Includes timeouts for long-running operations | |
| - **Memory Management**: Handles large datasets and visualization efficiently | |
| - **Unicode Handling**: Cleans problematic characters that might cause encoding issues | |
| ### Visualization Integration | |
| All visualizations are standardized to Plotly format: | |
| - Consistent styling and color schemes | |
| - Interactive features (zoom, pan, hover) | |
| - Accessibility compliance (colorblind-friendly palettes) | |
| - Export capabilities for reports | |
| - Responsive design for different screen sizes | |
| --- | |
| ## Frontend Integration | |
| The deep analysis system includes React components for: | |
| - **DeepAnalysisSidebar**: Main interface for starting and managing analyses | |
| - **NewAnalysisForm**: Form for initiating new deep analyses | |
| - **CurrentAnalysisView**: Real-time progress tracking during analysis | |
| - **HistoryView**: Browse and access historical analysis reports | |
| - **AnalysisStep**: Individual step progress visualization | |
| The frontend integrates with the streaming API to provide real-time feedback and uses the user's active template configuration for personalized analysis capabilities. | |
| ## Credit and Cost Tracking | |
| The system tracks detailed usage metrics: | |
| - **Credits Consumed**: Number of credits deducted from user account | |
| - **Token Usage**: Total tokens used across all model calls | |
| - **Estimated Cost**: Dollar cost estimate based on model pricing | |
| - **Model Information**: Provider and model name used for analysis | |
| - **Execution Time**: Duration of analysis for performance monitoring | |
| This information helps users understand resource consumption and optimize their analysis strategies. |