google-search-aj 1.0.6
Fast, lightweight Google search scraper with stealth mode
Requires Python: >=3.8
Keywords: google, search, scraper, web-scraping, playwright, automation

# Google Search Scraper

[![PyPI version](https://badge.fury.io/py/google-search-aj.svg)](https://badge.fury.io/py/google-search-aj)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
[![Python 3.8+](https://img.shields.io/badge/python-3.8+-blue.svg)](https://www.python.org/downloads/)

A fast, lightweight, and easy-to-use Python package for scraping Google search results with built-in stealth mode to avoid detection.

## ✨ Features

- 🚀 **Fast**: Optimized for speed with minimal overhead
- 🥷 **Stealth Mode**: Built-in anti-detection features
- 🎯 **Simple API**: Easy to use for both beginners and experts
- 📦 **Zero Config**: Playwright browser auto-installs on package installation
- 🔧 **Flexible**: Highly configurable with sensible defaults
- 💻 **CLI Support**: Use from command line or as a Python module
- 🎨 **Multiple Output Formats**: JSON and text output supported
- 📄 **Content Extraction**: Extract and analyze full page content from search results
- 💾 **Auto-Save**: Automatically save results to file with full content
- 🧹 **Clean Text**: Intelligent HTML parsing and text extraction

## 📦 Installation

```bash
pip install google-search-aj
```

The package will automatically install Playwright and download the Chromium browser during installation.

If automatic installation fails, manually install Playwright browsers:

```bash
playwright install chromium
```

## 🚀 Quick Start

### Python API

```python
from google_search_scraper import search

# Simple search
results = search("python tutorial")
print(results.urls)
# ['https://docs.python.org/3/tutorial/', 'https://www.w3schools.com/python/', ...]

# Access the direct answer (if available)
print(results.answer)
# 'Python is a high-level, general-purpose programming language...'

# Get more details
print(f"Found {results.total_results} results in {results.search_time:.2f} seconds")

# Extract full page content
results = search("machine learning", max_results=3, extract_content=True)
for content in results.contents:
    print(f"{content.title}: {content.word_count} words")
    print(f"Preview: {content.content[:200]}...")

# Auto-save to file with content extraction
results = search(
    "python tutorial", 
    extract_content=True, 
    save_to_file=True,  # Auto-save enabled
    output_file="search_results.txt"  # Custom filename
)
# Results automatically saved with full content!
```

### Command Line

```bash
# Simple search
google-search "python tutorial"

# Limit results
google-search "best restaurants" --max-results 20

# Save to file
google-search "machine learning" --output results.txt

# JSON output
google-search "data science" --format json

# Run with visible browser (debugging)
google-search "web scraping" --visible
```

## 📖 Usage Examples

### Basic Usage

```python
from google_search_scraper import search

# Default: 10 results with answer extraction
results = search("artificial intelligence")

for i, url in enumerate(results.urls, 1):
    print(f"{i}. {url}")
```

### Advanced Usage

```python
from google_search_scraper import GoogleSearchScraper

# Create a scraper with custom settings
scraper = GoogleSearchScraper(
    max_results=20,          # Get more results
    timeout=60000,           # Increase timeout
    headless=False,          # Show browser (for debugging)
    stealth_mode=True,       # Enable anti-detection (default)
    extract_content=True     # Extract page content
)

# Perform search
results = scraper.search("python web scraping", extract_answer=True)

# Access results
print(f"Query: {results.query}")
print(f"Time: {results.search_time:.2f}s")
print(f"Answer: {results.answer}")
print(f"URLs: {len(results.urls)}")
print(f"Content extracted: {len(results.contents)} pages")

# Convert to dictionary
data = results.to_dict()
```

### Content Extraction

```python
from google_search_scraper import search

# Extract content from search results
results = search("machine learning tutorial", max_results=5, extract_content=True)

# Access extracted content
for content in results.contents:
    if not content.error:
        print(f"\nTitle: {content.title}")
        print(f"URL: {content.url}")
        print(f"Word Count: {content.word_count:,}")
        print(f"Content Preview: {content.content[:200]}...")
    else:
        print(f"Failed to extract: {content.url} - {content.error}")

# Auto-save to file
results = search(
    "python web scraping",
    extract_content=True,
    save_to_file=True,  # Auto-save enabled
    output_file="my_search.txt"
)
print(f"✓ Results saved with full content!")

# Or save manually
results = search("AI tutorial", extract_content=True)
results.save_to_file("ai_results.txt")
```

### Multiple Searches

```python
from google_search_scraper import search

queries = ["python", "javascript", "rust"]

for query in queries:
    results = search(query, max_results=5)
    print(f"\n{query}: {len(results.urls)} results")
    print(results.urls[0] if results.urls else "No results")
```

### Error Handling

```python
from google_search_scraper import search
from google_search_scraper.exceptions import (
    GoogleSearchError,
    RateLimitError,
    BrowserError,
    SearchTimeoutError
)

try:
    results = search("test query", timeout=10000)
except RateLimitError:
    print("Being rate limited by Google. Try again later.")
except SearchTimeoutError:
    print("Search timed out. Try increasing the timeout.")
except BrowserError:
    print("Browser failed to launch.")
except GoogleSearchError as e:
    print(f"Search failed: {e}")
```

### Batch Processing

```python
from google_search_scraper import search
import time

queries = [
    "machine learning",
    "deep learning",
    "neural networks"
]

all_results = []

for query in queries:
    print(f"Searching: {query}")
    results = search(query, max_results=15)
    all_results.append(results)
    
    # Be respectful - add delay between searches
    time.sleep(5)

# Process results
for result in all_results:
    print(f"\n{result.query}:")
    print(f"  - Answer: {result.answer[:100] if result.answer else 'N/A'}")
    print(f"  - URLs: {len(result.urls)}")
```

## 🎯 API Reference

### Main Functions

#### `search(query, max_results=10, extract_answer=True, extract_content=False, headless=True, timeout=30000, save_to_file=False, output_file="search_results.txt")`

Convenience function for quick searches.

**Parameters:**
- `query` (str): Search query string
- `max_results` (int): Maximum URLs to return (default: 10)
- `extract_answer` (bool): Extract Google's direct answer (default: True)
- `extract_content` (bool): Extract page content from URLs (default: False)
- `headless` (bool): Run browser in headless mode (default: True)
- `timeout` (int): Page load timeout in milliseconds (default: 30000)
- `save_to_file` (bool): Automatically save results to file (default: False)
- `output_file` (str): Name of the output file (default: search_results.txt)

**Returns:** `SearchResult` object

### Classes

#### `GoogleSearchScraper`

Main scraper class with configurable options.

```python
scraper = GoogleSearchScraper(
    max_results=10,
    timeout=30000,
    headless=True,
    stealth_mode=True,
    user_agent=None,
    extract_content=False
)
```

**Methods:**
- `search(query, extract_answer=True)`: Perform a search

#### `SearchResult`

Container for search results.

**Attributes:**
- `query` (str): The search query
- `answer` (str | None): Google's direct answer if available
- `urls` (List[str]): List of result URLs
- `total_results` (int): Number of URLs returned
- `search_time` (float): Time taken for search in seconds
- `timestamp` (float): Unix timestamp of search
- `contents` (List[PageContent]): Extracted page content (if extract_content=True)

**Methods:**
- `to_dict()`: Convert to dictionary
- `save_to_file(filename="search_results.txt")`: Save results to text file with full content

#### `PageContent`

Container for extracted page content.

**Attributes:**
- `url` (str): The page URL
- `title` (str | None): Page title
- `content` (str): Extracted clean text content
- `word_count` (int): Number of words in content
- `error` (str | None): Error message if extraction failed

### Exceptions

- `GoogleSearchError`: Base exception for all errors
- `RateLimitError`: Raised when rate limited by Google
- `BrowserError`: Raised when browser fails
- `SearchTimeoutError`: Raised when search times out
- `NoResultsError`: Raised when no results found

## 🎛️ CLI Reference

```
usage: google-search [-h] [-n N] [--no-answer] [--visible] [--timeout MS]
                     [-o FILE] [-f {text,json}] [-v] [-q]
                     [query]

positional arguments:
  query                 Search query (if not provided, enters interactive mode)

optional arguments:
  -h, --help            show this help message and exit
  -n N, --max-results N
                        Maximum number of results to return (default: 10)
  --no-answer           Skip extracting Google's direct answer
  --visible             Run browser in visible mode (for debugging)
  --timeout MS          Page load timeout in milliseconds (default: 30000)
  -o FILE, --output FILE
                        Save results to file
  -f {text,json}, --format {text,json}
                        Output format: text or json (default: text)
  -v, --version         show program's version number and exit
  -q, --quiet           Suppress all output except results
```

## ⚠️ Important Notes

### Rate Limiting

Google may rate limit or block requests if you make too many searches too quickly. To avoid this:

1. Add delays between searches (5-10 seconds recommended)
2. Use residential proxies for large-scale scraping
3. Respect robots.txt and Google's Terms of Service

### Legal Considerations

Web scraping may have legal implications. This tool is for educational and research purposes. Users are responsible for:

- Complying with Google's Terms of Service
- Respecting robots.txt
- Following applicable laws and regulations
- Not using for commercial purposes without permission

### Detection

While this package includes stealth features, Google continuously updates their detection methods. If you're being blocked:

1. Increase delays between requests
2. Use `headless=False` to run in visible mode
3. Consider using residential proxies
4. Rotate user agents

## 🛠️ Development

### Setup Development Environment

```bash
# Clone the repository
git clone https://github.com/Aaditya17032002/google-search.git
cd google-search

# Create virtual environment
python -m venv venv
source venv/bin/activate  # On Windows: venv\Scripts\activate

# Install in development mode
pip install -e ".[dev]"

# Install Playwright browsers
playwright install chromium
```

### Running Tests

```bash
pytest tests/
```

### Code Formatting

```bash
black google_search_scraper/
flake8 google_search_scraper/
```

## 🤝 Contributing

Contributions are welcome! Please feel free to submit a Pull Request.

1. Fork the repository
2. Create your feature branch (`git checkout -b feature/AmazingFeature`)
3. Commit your changes (`git commit -m 'Add some AmazingFeature'`)
4. Push to the branch (`git push origin feature/AmazingFeature`)
5. Open a Pull Request

## 📝 License

This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.

## 🙏 Acknowledgments

- Built with [Playwright](https://playwright.dev/)
- Inspired by the need for a simple, reliable Google search scraper

## 📧 Support

If you encounter any issues or have questions:

- Open an issue on [GitHub](https://github.com/Aaditya17032002/google-search/issues)
- Check existing issues for solutions

## 🔄 Changelog

### v1.0.4 (2024-11-03)
- **Auto-save to file**: Save results with full content automatically
- Manual save with `save_to_file()` method
- Comprehensive file format with all data

### v1.0.3 (2024-11-03)
- **Content extraction**: Extract full page content from URLs
- BeautifulSoup4 integration for HTML parsing
- Async content extraction for better performance
- PageContent dataclass with title, content, word count

### v1.0.0 (2024-11-03)
- Initial release
- Fast Google search scraping
- Stealth mode with anti-detection
- CLI and Python API
- Auto-installation of Playwright
- JSON and text output formats

See [CHANGELOG.md](CHANGELOG.md) for full version history.

---

Made with ❤️ by developers, for developers