orman-news-reader 0.3.4
Extracts the main content from webpages and generates clean, reader-friendly HTML
License: MIT
Requires Python: <4.0,>=3.10
Keywords: web, reader, readability, html, content-extraction, google-news

# Web Reader Mode

A Python package that extracts the main content (text and images) from a webpage, similar to iPhone's browser reader mode.

## Features

- Extracts the main article content from a webpage
- Removes ads, navigation, and other distractions
- Downloads and saves images locally
- Outputs content in plain text or JSON format
- Generates clean, reader-friendly HTML pages

## Installation

### Using Poetry (Recommended)

1. Clone this repository:
   ```
   git clone <repository-url>
   cd <repository-directory>
   ```

2. Install with Poetry:
   ```
   poetry install
   ```

3. Alternatively, install from PyPI:
   ```
   pip install orman-news-reader
   ```

## Usage

### Using Poetry

```
poetry run orman-web-reader [command] [options]
```

Available commands:
- `reader`: Extract content from a webpage
- `html`: Generate a clean HTML page from a webpage
- `google`: Extract content from Google News RSS articles

### Reader Mode

Extract content from a webpage:
```
poetry run orman-web-reader reader https://example.com/article
```

Save images to a specific directory:
```
poetry run orman-web-reader reader https://example.com/article --output-dir images
```

Output in JSON format:
```
poetry run orman-web-reader reader https://example.com/article --json
```

Full options:
```
poetry run orman-web-reader reader --help
```

### Google News RSS Extractor

Extract content from Google News RSS articles with redirect handling:
```
poetry run orman-web-reader google https://news.google.com/rss/articles/[article-id]
```

Output in JSON format:
```
poetry run orman-web-reader google https://news.google.com/rss/articles/[article-id] --json
```

Save images to a specific directory:
```
poetry run orman-web-reader google https://news.google.com/rss/articles/[article-id] --output-dir images
```

Full options:
```
poetry run orman-web-reader google --help
```

### HTML Generator

Generate a clean HTML page from a webpage:
```
poetry run orman-web-reader html https://example.com/article
```

Specify output file and image directory:
```
poetry run orman-web-reader html https://example.com/article --output-file my_article.html --image-dir my_images
```

Process a Google News RSS article with redirect handling:
```
poetry run orman-web-reader html https://news.google.com/rss/articles/[article-id] --google-news
```

Use a custom CSS file:
```
poetry run orman-web-reader html https://example.com/article --css-file custom.css
```

Full options:
```
poetry run orman-web-reader html --help
```

### Direct Script Execution

You can also use the individual scripts directly:

```
poetry run orman-reader-mode https://example.com/article
poetry run orman-html-generator https://example.com/article
```

### Using as a Module

You can use the reader mode as a module in your own Python scripts:

```python
from orman_news_reader import extract_content
from orman_news_reader.html_generator import generate_html
from orman_news_reader.google_rss_extractor import extract_google_news_content

# Extract content from a regular URL
content = extract_content("https://example.com/article", "images")

# Access the extracted content
title = content['title']
paragraphs = content['text']
images = content['images']

# Generate HTML from the content
generate_html(content, "output.html")

# Extract content from a Google News RSS URL
google_content = extract_google_news_content("https://news.google.com/rss/articles/[article-id]", "images")

# Access Google News content with banner image
title = google_content['title']
banner_image = google_content.get('banner_image_url')
content_elements = google_content['content_elements']
```

## Example

```
poetry run orman-web-reader html https://example.com/article --output-file article.html
```

This will:
1. Extract the main content from the article
2. Save any images to the 'images' directory
3. Generate a clean HTML file with the article content
4. Apply a responsive design that works well on all devices

## How It Works

The package uses:
- `requests` to fetch the webpage
- `readability-lxml` to extract the main content
- `BeautifulSoup` to parse the HTML and extract text and images
- `Pillow` for image processing
- `Selenium` for handling JavaScript redirects in Google News RSS articles

### Standard Reader Mode
The standard reader mode extracts content directly from the provided URL using readability algorithms.

### Google News RSS Extractor
The Google News RSS extractor:
1. Uses Selenium WebDriver to follow redirects from Google News URLs to the actual article
2. Extracts high-quality banner images for carousels from Open Graph tags, Twitter cards, or featured images
3. Processes the article content using the standard reader mode
4. Returns a structured object with the article title, content, and banner image URL

## Requirements

- Python 3.8+
- Chrome/Chromium browser (for Selenium when using Google News features)
- Dependencies are managed by Poetry

## Development

### Setting up the development environment

```
git clone <repository-url>
cd <repository-directory>
poetry install
```

### Running tests

```
poetry run pytest
```

### Building the package

```
poetry build
```

### Publishing to PyPI

```
poetry publish
```