web-search-agent 0.2.0
An intelligent web search agent that provides structured results
License: MIT License
Requires Python: >=3.11

# Web Search Agent

A modular Python package for intelligent web search that processes search results into structured, section-based information.

## Overview

Web Search Agent is an infrastructure tool that performs web searches based on a given topic, processes the results using AI, and returns structured information organized into relevant sections. It helps users/developers quickly gather and organize comprehensive information on specific topics.

## Features

- Topic-based intelligent web searching
- AI-powered query generation for comprehensive coverage
- Automatic organization of search results into logical sections
- Section-specific sub-queries for in-depth analysis
- Structured JSON output for easy processing
- Configurable search parameters and model settings
- Support for multiple search APIs (Tavily, Exa)

## Installation

### Prerequisites

- Python 3.11 or higher
- OpenAI API key
- Search API key (Tavily, Exa, etc.)

### Installation Steps

```bash
# Create and activate a virtual environment (optional but recommended)
conda create -n web-search-agent python=3.11.11
conda activate web-search-agent

# Install from PyPI
pip install web-search-agent

# Or install from source
git clone https://github.com/leepokai/web-search-agent.git
cd web-search-agent
pip install -e .

# Set up environment variables
cp .env.example .env
# Edit .env file to add your API keys
```

## Usage

### Basic Usage

```python
import asyncio
from web_search_agent import WebSearchAgent, WebSearchConfig

async def main():
    # Create a default agent
    agent = WebSearchAgent()
    
    # Search for a topic
    result = await agent.search("Artificial Intelligence ethics")
    
    # Process the results
    print(f"Topic: {result.title}")
    print(f"Found {len(result.sections)} sections")
    
    for section in result.sections:
        print(f"- {section.title}: {section.description[:100]}...")

# Run the async function
asyncio.run(main())
```

### Advanced Usage

```python
import asyncio
from web_search_agent import WebSearchAgent, WebSearchConfig, search_multiple_topics

async def research_project():
    # Custom configuration
    config = WebSearchConfig(
        planner_model="o1",
        initial_queries_count=3,
        max_sections=5,
        search_api="tavily",
        search_api_config={"include_raw_content": True, "max_results": 3}
    )
    
    # Search multiple topics
    topics = [
        "Renewable energy advancements",
        "Future of remote work"
    ]
    
    results = await search_multiple_topics(
        topics,
        config=config,
        verbose=True,
        save_output=True,
        output_dir="research_output"
    )
    
    # Process results
    for result in results:
        if hasattr(result, 'title'):  # Check if valid result
            print(f"\nTopic: {result.title}")
            for section in result.sections:
                print(f"- {section.title}")

# Run the async function
if __name__ == "__main__":
    asyncio.run(research_project())
```

### Command Line Usage

```bash
# Basic usage
web-search-agent

```

## API Reference

### Main Classes

- `WebSearchAgent`: Core class for performing searches and processing results
- `WebSearchConfig`: Configuration class for customizing search behavior
- `WebSearchResult`: Result container with all search data and sections

### Key Functions

- `search_topic(topic, config=None, verbose=False)`: Search for a single topic
- `search_multiple_topics(topics, config=None, verbose=False, save_output=False)`: Search for multiple topics

## Configuration Options

The `WebSearchConfig` class provides the following configuration options:

```python
WebSearchConfig(
    # LLM Configuration
    llm_provider="openai",         # LLM provider (currently supports "openai")
    planner_model="o1",            # Model for planning and query generation
    
    # Search Configuration
    search_api="tavily",           # Search API (tavily, exa, etc.)
    search_api_config={            # Additional search API configuration
        "include_raw_content": True,
        "max_results": 3
    },
    
    # Query Generation
    initial_queries_count=3,       # Number of initial search queries
    section_queries_count=2,       # Number of search queries per section
    
    # Control Parameters
    max_sections=5,                # Maximum number of sections to generate
    must_cover_section_title="..." # Prompt for must-cover sections
)
```

## Output Format

The tool outputs structured data with the following main fields:

- `title`: Main search topic
- `initial_queries`: List of initial search queries
- `initial_responses`: Search results for each initial query
- `sections`: List of generated sections
  - Each section contains: title, description, search queries and responses

## Example Output

```json
{
  "title": "Artificial Intelligence Ethics",
  "initial_queries": [...],
  "initial_responses": [...],
  "sections": [
    {
      "title": "Privacy concerns in AI systems",
      "description": "An analysis of how AI technologies impact personal privacy...",
      "search_queries": [...],
      "search_responses": [...]
    },
    ...
  ]
}
```

## License

This project is licensed under the MIT License - see the LICENSE file for details.

## Contributing

Contributions are welcome! Please feel free to submit a Pull Request.