dingocrawl 0.3.0
A web crawling solution with FastAPI service, ModernBERT-based QA/NLI, Granite model for query expansion, Dempster-Shafer scoring, and MCTS-based query exploration with fact tables
License: MIT
Requires Python: >=3.8

# DingoCrawl

DingoCrawl is a powerful web crawling solution with a FastAPI service that integrates search, HTML-to-Markdown conversion, ModernBERT-based Question Answering (QA) and Natural Language Inference (NLI) scoring, and Granite model for query expansion and Dempster-Shafer claim scoring.

## Features

- **FastAPI Service**: Hot-loaded models for fast inference
- **Search**: Uses `duckduckgo-search` to find relevant URLs
- **Crawling**: Fetches HTML and converts it to Markdown using a custom rule-based converter
- **Bot Evasion**: Implements User-Agent rotation and realistic browser headers
- **QA**: Uses ModernBERT-based models for extractive QA
- **NLI**: Uses NLI models to score the entailment of answers
- **Query Expansion**: Uses IBM Granite 4.0-H-350M model for MECE query expansion with SERP analysis
- **Dempster-Shafer Scoring**: Scores claims against facts using Granite model
- **MCTS Exploration**: Monte Carlo Tree Search for query exploration with fact table generation
- **GPU Support**: Automatic GPU detection and usage (CUDA > MPS > CPU)

## Installation

```bash
pip install dingocrawl
```

## Quick Start

### Start the FastAPI Service

```bash
dingocrawl start
```

The service will be available at `http://127.0.0.1:8000`

### Stop the Service

```bash
dingocrawl stop
```

### API Documentation

Once running, visit:
- Swagger UI: http://127.0.0.1:8000/docs
- ReDoc: http://127.0.0.1:8000/redoc

## API Endpoints

### Health Check
```bash
curl http://127.0.0.1:8000/health
```

### Query Expansion
```bash
curl -X POST http://127.0.0.1:8000/expand \
  -H "Content-Type: application/json" \
  -d '{"query": "machine learning", "num_queries": 5}'
```

### Dempster-Shafer Claim Scoring
```bash
curl -X POST http://127.0.0.1:8000/score \
  -H "Content-Type: application/json" \
  -d '{
    "claim": "Python is the best language for beginners",
    "facts": [
      "Python has simple syntax",
      "Python has extensive documentation"
    ]
  }'
```

### Question Answering
```bash
curl -X POST http://127.0.0.1:8000/qa \
  -H "Content-Type: application/json" \
  -d '{
    "context": "Python is a programming language created by Guido van Rossum.",
    "question": "Who created Python?"
  }'
```

### Full Pipeline
```bash
curl -X POST http://127.0.0.1:8000/process \
  -H "Content-Type: application/json" \
  -d '{"query": "What is Python?", "max_results": 3}'
```

### MCTS Query Exploration

The MCTS (Monte Carlo Tree Search) endpoint explores queries through iterative expansion, evidence collection, and fact table creation with Dempster-Shafer scoring.

#### Basic Usage (Delta Entropy Stopping)

Stops when entropy change is less than 0.05 (default):

```bash
curl -X POST http://127.0.0.1:8000/mcts \
  -H "Content-Type: application/json" \
  -d '{"query": "What is machine learning?"}'
```

#### Custom Delta Entropy Threshold

```bash
curl -X POST http://127.0.0.1:8000/mcts \
  -H "Content-Type: application/json" \
  -d '{
    "query": "What is machine learning?",
    "delta_entropy": 0.1
  }'
```

#### Budget-Based Stopping

Stops when search/crawl operations reach the budget limit (max 1000):

```bash
curl -X POST http://127.0.0.1:8000/mcts \
  -H "Content-Type: application/json" \
  -d '{
    "query": "What is machine learning?",
    "budget": 10
  }'
```

#### Custom Expansion Prompt

```bash
curl -X POST http://127.0.0.1:8000/mcts \
  -H "Content-Type: application/json" \
  -d '{
    "query": "How does Tesla optimize operations?",
    "budget": 5,
    "custom_prompt": "Generate questions using Porter's Five Forces framework"
  }'
```

#### Response Structure

The MCTS endpoint returns:

```json
{
  "fact_tables": [
    {
      "fact": "Extracted fact text",
      "markdown": "Full markdown from crawled page",
      "url": "Source URL",
      "answer": "Answer to the node's question using ModernBert QA",
      "supports": 0.7,
      "contradicts": 0.1,
      "ignorance": 0.2,
      "node_id": "query-node-id"
    }
  ],
  "query_nodes": [
    {
      "id": "node-id",
      "question": "Query question",
      "parent_id": "parent-id or null",
      "level": 0,
      "children": [],
      "is_expanded": true,
      "entropy": 0.5,
      "visit_count": 1,
      "pignistic_probabilities": {}
    }
  ],
  "final_entropy": 0.45,
  "iterations": 5,
  "search_count": 8
}
```

#### Stopping Criteria

- **Delta Entropy** (default: 0.05): Stops when entropy change over last 3 iterations < threshold
- **Budget** (max: 1000): Stops when search/crawl operations reach limit
- **Mutually Exclusive**: Only one stopping criterion can be specified

#### Fact Table Fields

Each fact table entry contains:
- **fact**: Extracted fact text from markdown
- **markdown**: Full markdown content from crawled page
- **url**: Source URL of the page
- **answer**: Answer to the node's question generated using ModernBert QA
- **supports/contradicts/ignorance**: Dempster-Shafer belief scores
- **node_id**: ID of the query node this fact belongs to

## Command Line Usage

### Legacy Query Mode (Backward Compatible)
```bash
dingocrawl --query "What is the capital of France?"
```

## Python API

```python
from dingocrawl import search_web, fetch_page, html_to_markdown, ModernBertQA

# Search
urls = search_web("Python programming")

# Crawl
html = fetch_page(urls[0])
markdown = html_to_markdown(html)

# QA
qa = ModernBertQA()
answer = qa.answer(markdown, "What is Python?")
print(answer)
```

## Models

- **ModernBertQA**: Question answering model (deepset/roberta-base-squad2)
- **ModernBertNLI**: Natural language inference model (cross-encoder/nli-deberta-v3-base)
- **Granite Query Expander**: IBM Granite 4.0-H-350M for query expansion and claim scoring

All models are hot-loaded at service startup for fast inference. GPU is automatically detected and used when available (CUDA > MPS > CPU).

## MCTS Algorithm

The MCTS (Monte Carlo Tree Search) implementation:

1. **Expansion**: Starts with root query, expands nodes using MECE query expansion
2. **Evidence Collection**: Searches and crawls pages for each expanded node
3. **Fact Extraction**: Extracts facts from markdown content
4. **QA Generation**: Generates answers using ModernBert QA for each fact table entry
5. **Dempster-Shafer Scoring**: Scores facts against node questions
6. **Entropy Calculation**: Uses Dempster-Shafer combination to calculate node-level entropy
7. **Stopping**: Stops when delta entropy < threshold or budget exhausted

The algorithm builds a tree of query nodes, where each node represents a question. Facts are collected and scored, with entropy calculated using Dempster-Shafer theory to determine when exploration has reached equilibrium.

## License

MIT