mcp-websource 0.1.3
Modular Content Pipeline: trend detection, relevance scoring, fact verification, and knowledge enrichment. Python library; no web search.
Requires Python: >=3.10
Keywords: trends, fact-check, web-scraping, research, nlp, pipeline

# MCP Websource + Trend Intelligence System

**Modular Content Pipeline (MCP)** — open-source Python library for multi-platform trend detection, relevance scoring, fact verification, and knowledge enrichment. No web search; content from trends or user-provided data.

**Installing the library does not run any pipeline or server.** Use the classes or convenience functions in your code; the API server (`run.py`) is optional and must be started explicitly.

---

## Features

- **Multi-Platform Trend Detector**: Hacker News, Reddit, GitHub trending; trend velocity; topic signals
- **Relevance Engine**: Embedding similarity, freshness decay, authority scoring
- **Fact Check Engine**: Verification from provided evidence items; confidence and evidence links
- **Knowledge Enrichment**: Wikipedia summaries, NER (spaCy), entity linking, topic extraction
- **Ranking & Aggregation**: Combined score (relevance + credibility + trend + freshness)
- **Content Fetcher**: Visit URLs, extract main text, summarize
- **Storage**: Optional document store, Redis cache
- **REST API**: `/query`, `/trends`, `/fact-check`, `/research`, `/health`
- **Library API**: Use each module separately (trends, fact-check, fetch URLs, enrich, full pipeline) — see [docs/LIBRARY.md](docs/LIBRARY.md)

---

## Library usage (class-based)

Use the main class **`MCPWebsource`**; nothing runs until you call a method:

```python
import asyncio
from mcp_websource import MCPWebsource

client = MCPWebsource()
trends = asyncio.run(client.get_trends())
out = asyncio.run(client.run_pipeline("AI regulation"))
```

Or use convenience functions (they use a default `MCPWebsource()` instance):

```python
from mcp_websource import get_trends, fact_check, fetch_and_summarize_urls, enrich, run_pipeline
import asyncio

trends = asyncio.run(get_trends())
confidence, evidence = asyncio.run(fact_check("The Earth is round.", evidence_items=[]))
results = asyncio.run(fetch_and_summarize_urls(["https://example.com/article"]))
enriched = enrich(item)
out = asyncio.run(run_pipeline("AI regulation"))
```

You can also use the underlying classes directly: `TrendDetector`, `FactCheckEngine`, `ContentFetcher`, `EnrichmentEngine`, `PipelineOrchestrator`. See [docs/LIBRARY.md](docs/LIBRARY.md) and [examples/library_usage.py](examples/library_usage.py).

---

## Architecture

```
Input (query) → Trend Detection → Relevance → Rank → Fetch content → Fact check (from sources) → Enrichment → Storage → API Output
```

See [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md) for details.

---

## Install (library only — no pipeline or server runs)

**From PyPI** (after you publish):

```bash
pip install mcp-websource
# Optional: ML/embeddings, vector store, or everything
pip install mcp-websource[ml]
pip install mcp-websource[all]
```

Install only adds the package; no pipeline or server is started. Use in code:

```python
from mcp_websource import MCPWebsource
client = MCPWebsource()
# ... call client.get_trends(), client.run_pipeline(), etc.
```

**Publishing to PyPI:** Run `python scripts/upload_pypi.py` (or `scripts/upload_pypi.ps1` on Windows). See [docs/PUBLISHING.md](docs/PUBLISHING.md) for details.

---


### 2. Configure (optional)

Copy `.env.example` to `.env` and set:

- `REDIS_URL` — for caching (optional)
- `LOG_LEVEL` — e.g. DEBUG
- `INCLUDE_REDDIT_TRENDS` — set to `false` in Colab/cloud if Reddit returns 403

### 3. Run API

```bash
python run.py
```

- API: http://127.0.0.1:8000  
- Docs: http://127.0.0.1:8000/docs  

### 4. Example requests

```bash
curl "http://127.0.0.1:8000/health"
curl "http://127.0.0.1:8000/query?q=AI+regulation"
curl "http://127.0.0.1:8000/trends"
```

See [examples/api_usage.md](examples/api_usage.md) for full examples.

---

## Output format (research-grade)

Structured JSON per query:

```json
{
  "topic": "string",
  "summary": "string",
  "sources": [{"title", "url", "relevance_score", "credibility_score"}],
  "trend_score": 0.0,
  "relevance_score": 0.0,
  "fact_confidence": 0.0,
  "entities": [{"text", "label", "start", "end"}],
  "timestamp": "ISO8601",
  "evidence_links": [{"url", "title", "snippet", "confidence"}],
  "metadata": {}
}
```

Sample: [examples/sample_output.json](examples/sample_output.json).

---

## Tests

```bash
pytest tests/ -v
```

---

## Project structure

```
mcp for websource/
├── mcp_websource/
│   ├── config/          # Settings (env)
│   ├── core/            # Models, logging
│   ├── modules/
│   │   ├── trend_detector/
│   │   ├── relevance_engine/
│   │   ├── fact_check/
│   │   ├── enrichment/
│   │   ├── ranking/
│   │   ├── content_fetcher/
│   │   └── storage/
│   ├── pipeline/        # Orchestrator
│   └── api/             # FastAPI app
├── tests/
├── docs/
├── examples/
├── run.py
├── requirements.txt
└── README.md
```

---

## Topics coverage

The system supports trend detection and research across:

Technology, Science, News, Politics, Social media, Psychology, Finance, Business, Geopolitics, Global events, Internet culture, Research papers, Startups, AI/ML, Economics, Cybersecurity, Health trends, Consumer behavior, Wikipedia knowledge graph, and general trending topics.

---

## License and attribution

Open-source friendly; replaceable components; no hardcoded secrets. Use at your own risk for production (rate limits, ToS of third-party APIs apply).