subject-researcher-mcp 0.2.0
Enhanced Subject Researcher MCP - Iterative target-driven research with multi-vertical search, quality metrics, and sophisticated claim mining
License: MIT
Requires Python: >=3.10
Keywords: ai, claim-mining, credibility-scoring, evidence-based, iterative-research, mcp, multi-vertical-search, quality-metrics, research, web-search

# 🚀 Enhanced Subject Researcher MCP

[![CI/CD Pipeline](https://github.com/your-org/subject-researcher-mcp/workflows/CI/CD%20Pipeline/badge.svg)](https://github.com/your-org/subject-researcher-mcp/actions)
[![Python 3.10+](https://img.shields.io/badge/python-3.10+-blue.svg)](https://www.python.org/downloads/)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
[![PyPI version](https://badge.fury.io/py/subject-researcher-mcp.svg)](https://badge.fury.io/py/subject-researcher-mcp)

> **Advanced iterative target-driven research with multi-vertical search, sophisticated claim mining, and evidence-based synthesis**

## ✨ Features

### 🎯 **Iterative Target-Driven Research**
- **Quality Meters**: Coverage, recency, novelty, agreement, contradictions tracking
- **Stop Criteria**: Configurable quality gates with automatic continuation logic
- **Stagnation Detection**: Automatic scope widening when research plateaus
- **Adaptive Queries**: Smart query generation based on iteration state

### 🔍 **Multi-Vertical Search Engine**
- **Real Web Search**: DuckDuckGo integration for actual web results
- **5 Search Verticals**: Web, News, Docs, Community, Academic sources
- **Fallback System**: Graceful degradation with high-quality synthetic results
- **Source Deduplication**: Cross-iteration URL tracking

### ⚗️ **Sophisticated Claim Mining**
- **Atomic Claims**: Extracts falsifiable, standalone statements
- **Metadata Extraction**: Units, measurements, caveats automatically detected
- **Independence Detection**: Cross-source validation and duplicate identification
- **Confidence Scoring**: Evidence-based claim reliability assessment

### 📊 **Enhanced Credibility Scoring**
- **Multi-Factor Analysis**: Domain authority, recency, content quality, independence
- **Independence Matrix**: Detects source relationships and potential bias
- **Transparency**: Detailed credibility breakdown for every source
- **Real-Time Updates**: Dynamic scoring based on cross-validation

### 📋 **Answer-First Synthesis**
- **Direct Answers**: Immediate response to research questions
- **Inline Citations**: Professional citation system with automatic numbering
- **Evidence Weighting**: Confidence scores based on source quality
- **Professional Reports**: Executive summaries with actionable recommendations

## 🚀 Quick Start

### 🐳 Recommended: Docker Installation (Easiest)

The easiest way to use the Enhanced Subject Researcher MCP is with Docker - no Python dependencies or environment setup required!

**Why Docker?**
- ✅ **Zero setup** - No Python environment configuration needed
- ✅ **Consistent** - Works the same across all systems (Windows, macOS, Linux)
- ✅ **Isolated** - No conflicts with your existing Python packages
- ✅ **Latest version** - Always get the most recent release automatically

```bash
# Pull the latest Docker image
docker pull elad12390/subject-researcher-mcp:latest

# Test the server (optional)
docker run --rm -i elad12390/subject-researcher-mcp:latest
```

### Alternative: Python Installation

```bash
pip install subject-researcher-mcp
```

### Basic Usage

```python
import asyncio
from subject_researcher_mcp import ResearchEngine, ResearchInputs

async def research_example():
    engine = ResearchEngine()
    
    # Configure research parameters
    inputs = ResearchInputs(
        subject="Python async programming best practices",
        objective="comprehensive_analysis",
        max_sources=15,
        constraints={
            "max_iterations": 3,
            "gate_thresholds": {
                "min_coverage": 0.7,
                "min_recency": 0.5,
                "novelty_threshold": 0.1,
                "max_contradictions": 0.3
            }
        }
    )
    
    # Execute iterative research
    report = await engine.conduct_iterative_research(inputs)
    
    print(f"Research completed: {len(report.sources)} sources, {len(report.claims)} claims")
    print(f"Confidence: {report.confidence:.1%}")
    print(f"Executive Summary: {report.executive_summary}")
    
    await engine.close()

# Run the research
asyncio.run(research_example())
```

## 🔧 Installation for AI Editors

### Claude Desktop

**🐳 Recommended: Docker Method (Easiest)**

1. **Pull the Docker image:**
   ```bash
   docker pull elad12390/subject-researcher-mcp:latest
   ```

2. **Configure Claude Desktop:**
   - Open Claude Desktop
   - Click the Claude menu → Settings
   - Go to Developer tab → Edit Config
   - Add this configuration:

   ```json
   {
     "mcpServers": {
       "subject-researcher": {
         "command": "docker",
         "args": [
           "run",
           "--rm",
           "-i",
           "elad12390/subject-researcher-mcp:latest"
         ],
         "env": {
           "GEMINI_API_KEY": "your-optional-gemini-api-key"
         }
       }
     }
   }
   ```

3. **Restart Claude Desktop** and look for the MCP server indicator (🔌) in the chat input.

**📦 Alternative: Python Method**

1. **Install the package:**
   ```bash
   pip install subject-researcher-mcp
   ```

2. **Configure Claude Desktop:**
   ```json
   {
     "mcpServers": {
       "subject-researcher": {
         "command": "python",
         "args": ["-m", "subject_researcher_mcp.server"],
         "env": {
           "GEMINI_API_KEY": "your-optional-gemini-api-key"
         }
       }
     }
   }
   ```

### Cursor IDE

**🐳 Recommended: Docker Method (Easiest)**

1. **Pull the Docker image:**
   ```bash
   docker pull elad12390/subject-researcher-mcp:latest
   ```

2. **Configure Cursor:**
   - Create `.cursor/mcp.json` in your project root (or `~/.cursor/mcp.json` for global access)
   - Add this configuration:

   ```json
   {
     "mcpServers": {
       "subject-researcher": {
         "command": "docker",
         "args": [
           "run",
           "--rm", 
           "-i",
           "elad12390/subject-researcher-mcp:latest"
         ],
         "enabled": true,
         "env": {
           "GEMINI_API_KEY": "your-optional-gemini-api-key"
         }
       }
     }
   }
   ```

3. **Usage in Cursor:**
   - Open the Composer Agent
   - MCP tools will be listed under "Available Tools"
   - Ask for research using natural language

**📦 Alternative: Python Method**

1. **Install the package:**
   ```bash
   pip install subject-researcher-mcp
   ```

2. **Configure Cursor:**
   ```json
   {
     "mcpServers": {
       "subject-researcher": {
         "command": "python",
         "args": ["-m", "subject_researcher_mcp.server"],
         "enabled": true,
         "env": {
           "GEMINI_API_KEY": "your-optional-gemini-api-key"
         }
       }
     }
   }
   ```

### Claude Code

**🚀 Command-Line Method (Easiest)**

```bash
# Using Docker (Recommended)
claude mcp add subject-researcher --env GEMINI_API_KEY=your-optional-key \
  -- docker run --rm -i elad12390/subject-researcher-mcp:latest

# Or using Python
claude mcp add subject-researcher --env GEMINI_API_KEY=your-optional-key \
  -- python -m subject_researcher_mcp.server
```

**🔧 Manual Configuration**

1. **Pull the Docker image:**
   ```bash
   docker pull elad12390/subject-researcher-mcp:latest
   ```

2. **Add manually via JSON:**
   ```bash
   claude mcp add-json subject-researcher '{
     "type":"stdio",
     "command":"docker",
     "args":["run","--rm","-i","elad12390/subject-researcher-mcp:latest"],
     "env":{"GEMINI_API_KEY":"your-optional-key"}
   }'
   ```

### OpenCode

**🐳 Recommended: Docker Method (Easiest)**

1. **Pull the Docker image:**
   ```bash
   docker pull elad12390/subject-researcher-mcp:latest
   ```

2. **Configure OpenCode:**
   - In your project directory, edit `opencode.json`
   - Add this to the configuration:

   ```json
   {
     "$schema": "https://opencode.ai/config.json",
     "mcp": {
       "subject-researcher": {
         "type": "local",
         "command": [
           "docker",
           "run",
           "--rm",
           "-i", 
           "elad12390/subject-researcher-mcp:latest"
         ],
         "enabled": true,
         "environment": {
           "GEMINI_API_KEY": "your-optional-gemini-api-key"
         }
       }
     }
   }
   ```

3. **Usage in OpenCode:**
   - MCP tools are automatically available to the LLM
   - Ask for research and OpenCode will use the tools as needed

**📦 Alternative: Python Method**

1. **Install the package:**
   ```bash
   pip install subject-researcher-mcp
   ```

2. **Configure OpenCode:**
   ```json
   {
     "$schema": "https://opencode.ai/config.json",
     "mcp": {
       "subject-researcher": {
         "type": "local",
         "command": ["python", "-m", "subject_researcher_mcp.server"],
         "enabled": true,
         "environment": {
           "GEMINI_API_KEY": "your-optional-gemini-api-key"
         }
       }
     }
   }
   ```

## 📖 MCP Server Usage

Once configured in your AI editor, you can use natural language to request research:

**Example requests:**
- "Research the latest developments in quantum computing applications"
- "Analyze current best practices for microservices architecture"
- "Investigate recent security vulnerabilities in popular Python packages"

The MCP server provides these tools:
- `conduct_iterative_research` - Full 11-phase research methodology
- `conduct_research` - Basic multi-source research
- `analyze_research_quality` - Quality assessment of research results

## 🔑 Environment Variables

The Subject Researcher MCP supports the following optional environment variables:

| Variable | Description | Required | Default |
|----------|-------------|----------|---------|
| `GEMINI_API_KEY` | Google Gemini API key for enhanced analysis and synthesis | No | Not used |

**Note:** The research engine works fully without any API keys, using free search APIs. The Gemini API key is only used for optional enhanced analysis features.

### Direct MCP Server Usage

```bash
# Start the MCP server
python -m subject_researcher_mcp.server

# Or using Docker
docker run -p 8000:8000 your-org/subject-researcher-mcp:latest
```

## 🛠️ Development

### Prerequisites

- Python 3.10+
- Git

### Setup

```bash
# Clone the repository
git clone https://github.com/your-org/subject-researcher-mcp.git
cd subject-researcher-mcp

# Install in development mode
pip install -e .

# Install development dependencies
pip install pytest pytest-asyncio ruff build
```

### Running Tests

```bash
# Run all tests
pytest tests/ -v

# Run E2E tests
pytest tests/test_real_mcp_e2e.py -v

# Run quick validation
python -c "from src.subject_researcher_mcp.research_engine import ResearchEngine; print('✅ Validation passed')"
```

### Code Quality

```bash
# Lint code
ruff check src/ tests/

# Format code
ruff format src/ tests/

# Type checking (if using mypy)
mypy src/
```

## 📖 API Reference

### ResearchEngine

Main class for conducting iterative research.

```python
class ResearchEngine:
    def __init__(self, gemini_api_key: Optional[str] = None)
    
    async def conduct_iterative_research(self, inputs: ResearchInputs) -> ResearchReport
    async def conduct_research(self, inputs: ResearchInputs) -> ResearchReport  # Legacy method
```

### ResearchInputs

Configuration for research execution.

```python
@dataclass
class ResearchInputs:
    subject: str
    objective: str = "comprehensive_analysis"  # or "best_options", "decision_support"
    depth: str = "standard"  # "fast", "standard", "deep"
    max_sources: int = 50
    recency_months: int = 18
    constraints: Dict[str, Any] = field(default_factory=dict)
```

### Quality Gates Configuration

```python
gate_thresholds = {
    "min_coverage": 0.7,        # Minimum topic coverage (0-1)
    "min_recency": 0.5,         # Minimum source freshness (0-1)
    "novelty_threshold": 0.1,   # Minimum new info rate (0-1)
    "max_contradictions": 0.3   # Maximum contradiction level (0-1)
}
```

## 🔧 Configuration

### Environment Variables

```bash
# Optional: Gemini API key for enhanced AI analysis
export GEMINI_API_KEY="your-gemini-api-key"

# Optional: Custom configuration
export RESEARCH_MAX_ITERATIONS=5
export RESEARCH_TIMEOUT=300
```

### MCP Client Configuration

```json
{
  "mcpServers": {
    "subject-researcher": {
      "command": "python",
      "args": ["-m", "subject_researcher_mcp.server"],
      "env": {
        "GEMINI_API_KEY": "your-key-here"
      }
    }
  }
}
```

## 🏗️ Architecture

### Research Methodology

The Enhanced Subject Researcher implements an 11-phase iterative methodology:

1. **Plan** - Generate research questions and hypotheses
2. **Query Design** - Create adaptive search queries
3. **Harvest** - Multi-vertical search execution
4. **Triage** - Source quality filtering
5. **Claim Mining** - Atomic claim extraction
6. **Cluster & Triangulate** - Cross-source validation
7. **Evaluate Credibility** - Enhanced scoring with independence matrix
8. **Topic Logic** - Domain-specific analysis (for "best X" queries)
9. **Synthesize** - Answer-first report generation
10. **Self-Critique** - Gap identification and quality assessment
11. **Package & Verify** - Final report assembly and validation

### Quality Metrics

- **Coverage**: How comprehensively the topic has been researched
- **Recency**: Average age and freshness of sources
- **Novelty**: Rate of new information discovery per iteration
- **Agreement**: Level of consensus across sources
- **Contradictions**: Amount of conflicting information found

## 🔒 Security

### Data Privacy
- No personal data collection
- API keys handled securely
- Source URLs and content processed locally

### Security Scanning
- Automated dependency vulnerability scanning
- Code security analysis with Bandit
- Regular security updates

## 📊 Performance

### Benchmarks
- **Search Speed**: 2-5 real sources per iteration (15-45 seconds)
- **Claim Extraction**: 2-3 atomic claims per source
- **Memory Usage**: ~50-100MB for standard research
- **Accuracy**: 85%+ confidence scores in controlled tests

### Optimization Tips
- Use `depth="fast"` for quick research (2-3 iterations)
- Adjust `max_sources` based on thoroughness needs
- Configure `gate_thresholds` for different quality requirements

## 🤝 Contributing

We welcome contributions! Please see our [Contributing Guide](CONTRIBUTING.md) for details.

### Development Workflow

1. Fork the repository
2. Create a feature branch (`git checkout -b feature/amazing-feature`)
3. Make your changes
4. Add tests for new functionality
5. Ensure all tests pass (`pytest`)
6. Submit a pull request

### Code Standards

- Follow PEP 8 style guidelines
- Add type hints for all functions
- Include docstrings for public APIs
- Maintain test coverage above 80%

## 📝 License

This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.

## 🙏 Acknowledgments

- [MCP Protocol](https://github.com/modelcontextprotocol/python-sdk) for the foundation
- [DuckDuckGo](https://duckduckgo.com) for search capabilities
- [Wikipedia API](https://pypi.org/project/Wikipedia-API/) for reliable reference data
- Research methodology inspired by academic research best practices

## 📈 Roadmap

### v2.1.0 (Planned)
- [ ] Real-time research monitoring dashboard
- [ ] Advanced NLP for better claim extraction
- [ ] Integration with academic databases
- [ ] Research collaboration features

### v2.2.0 (Future)
- [ ] Machine learning for query optimization
- [ ] Multi-language research support
- [ ] Advanced visualization tools
- [ ] Research template system

## 📞 Support

- **Documentation**: [GitHub Wiki](https://github.com/your-org/subject-researcher-mcp/wiki)
- **Issues**: [GitHub Issues](https://github.com/your-org/subject-researcher-mcp/issues)
- **Discussions**: [GitHub Discussions](https://github.com/your-org/subject-researcher-mcp/discussions)
- **Email**: research-support@your-org.com

---

**Made with ❤️ by the Enhanced Subject Researcher Team**

> *"Transforming information chaos into evidence-based insights through intelligent automation"*