py-web-text-extractor 0.1.0
Library-wrapper for extracting text from Web Pages
License: MIT
Requires Python: >=3.14
Keywords: web, text, extraction, scraping, html, markdown, trafilatura, markitdown

# py-web-text-extractor

[![CI](https://github.com/sanyokkua/py_web_text_extractor/actions/workflows/ci.yml/badge.svg)](https://github.com/sanyokkua/py_web_text_extractor/actions)
[![PyPI version](https://badge.fury.io/py/py-web-text-extractor.svg)](https://pypi.org/project/py-web-text-extractor/)
[![Python 3.14+](https://img.shields.io/badge/python-3.14+-blue.svg)](https://www.python.org/downloads/)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)

A CLI tool and Python library to extract clean text content from web pages.

## Overview

`py-web-text-extractor` provides a simple interface for extracting the main text content from HTML documents. It can be used as a command-line tool for quick extractions or as a Python library for integration into other applications. The tool uses a fallback strategy, trying `markitdown` first and then `trafilatura` to ensure high reliability.

## Features

- **Dual Extractor Strategy**: Uses `markitdown` as the primary extractor and falls back to `trafilatura` for robustness.
- **CLI and Library Interface**: Can be used as a standalone command-line tool or as a Python library.
- **Error Handling Modes**: Supports a strict mode that raises exceptions on failure and a safe mode that returns an empty string.
- **Modern Python**: Fully typed with Python 3.14+ support.

## Prerequisites

- Python 3.14 or higher.

## Installation

You can install the package using `pip` or `uv`.

### Using pip

```bash
pip install py-web-text-extractor
```

### Using uv

```bash
uv add py-web-text-extractor
```

## Usage

The tool can be used via its command-line interface or as a Python library.

### Command-Line Interface (CLI)

The CLI is the quickest way to extract text from a URL.

**Basic Extraction:**

```bash
py-web-text-extractor https://example.com
```

**Safe Mode:**

In safe mode, the tool will return an empty string and exit gracefully if an error occurs.

```bash
py-web-text-extractor https://example.com --safe
```

**Verbose Mode:**

For troubleshooting, verbose mode provides detailed debug output.

```bash
py-web-text-extractor https://example.com --verbose
```

**CLI Exit Codes:**

| Code | Meaning                |
| ---- | ---------------------- |
| 0    | Success                |
| 1    | No text content found  |
| 2    | Invalid URL            |
| 3    | Text extraction failed |
| 4    | Unexpected error       |


### Python Library

For programmatic use, you can import the `ExtractorService`.

**Quick Start:**

```python
from py_web_text_extractor.service.extractor_service import ExtractorService
from py_web_text_extractor.exception.exceptions import TextExtractionError, UrlIsNotValidException

# Initialize the service
service = ExtractorService()

# Strict mode: raises an exception on failure
try:
    text = service.extract_text_from_page("https://example.com")
    print(text)
except UrlIsNotValidException:
    print("The provided URL is not valid.")
except TextExtractionError as e:
    print(f"Failed to extract text: {e}")

# Safe mode: returns an empty string on failure
text_safe = service.extract_text_from_page_safe("https://invalid-url")
if not text_safe:
    print("Extraction failed or no content found.")

```

## API Reference

### `ExtractorService`

The main class for handling text extraction.

**Methods:**

- **`extract_text_from_page(url: str) -> str`**: Extracts text from the given URL. Raises a `TextExtractionError` or `UrlIsNotValidException` on failure.
- **`extract_text_from_page_safe(url: str) -> str`**: Extracts text from the given URL. Returns an empty string on failure.

### Exceptions

The library uses a set of custom exceptions to allow for specific error handling.

- `TextExtractionError`: Base exception for the library.
- `UrlIsNotValidException`: Raised for invalid URL formats.
- `TextExtractionFailure`: Raised when all extraction attempts fail.
- `MarkItDownExtractionException`: Specific failure from the `markitdown` extractor.
- `TrafilaturaExtractionException`: Specific failure from the `trafilatura` extractor.


## Architecture

The service employs a fallback strategy to maximize reliability:
1.  It first attempts to extract content using `markitdown`.
2.  If `markitdown` fails (e.g., returns a blank string or raises an error), the service automatically retries the extraction using `trafilatura`.
3.  The first successful result is returned. If both extractors fail, an error is raised or an empty string is returned, depending on the mode.

## Testing

To run the test suite, first install the development dependencies and then run `pytest`.

> Patch pydub for Python 3.14 compatibility
> For Python 3.14 can be required patching of old dependency

```bash
python scripts/patch_pydub.py
```

```bash
# Install dev dependencies
uv pip install -e ".[dev]"

# Run tests
uv run pytest

# Run linters and type checkers
uv run ruff check .
uv run ruff format .
uv run mypy src/
```

## License

This project is licensed under the MIT License. See the [LICENSE](LICENSE) file for details.