codexcollector 0.2.0
A wrapper for common Python packages to build a text corpus with a single line of code.
License: MIT
Requires Python: >=3.10
Keywords: NLP, corpus building, document extraction, document parsing, docx, natural language processing, pdf, text extraction, text processing, web scraping

# CodexCollector

A wrapper for common Python packages to build a text corpus with a single line of code.

## Overview

CodexCollector provides a unified interface for document ingestion from both filesystem paths and web URLs. It automatically handles format detection, text extraction, metadata collection, and error logging, making it easy to build text corpora from diverse document sources.

**Key Capabilities:**
- Extract text from multiple document formats (Word, PowerPoint, PDF, plain text)
- Collect documents from local filesystems with recursive directory traversal
- Scrape and download documents from websites with configurable crawling depth
- Extract text directly from web pages (HTML content extraction)
- Automatic encoding detection for text files
- Metadata extraction (creation dates) where available
- Comprehensive error logging for failed ingestions
- Rate limiting and timeout controls for web requests
- File size validation and filtering

## Installation

### Requirements
- Python >=3.10
- Dependencies installed automatically via pip/uv

### From PyPI (Recommended)

```bash
# Using uv
uv add codexcollector

# Using pip
pip install codexcollector
```

### From Source

```bash
# Clone the repository
git clone https://github.com/NotAndroid37/codexcollector.git
cd codexcollector

# Using uv
uv sync

# Or using pip
pip install -e .
```

## Quick Start

### Basic Filesystem Collection

```python
from codexcollector import CodexCollector

# Initialize collector with default settings
collector = CodexCollector()

# Collect documents from a directory
corpus = collector.collect('/path/to/documents')

# Access collected documents (returns a list)
for document in corpus:
    print(f"File: {document['filename']}")
    print(f"Source: {document['source']}")
    print(f"Text length: {len(document['text'])} characters")
    print(f"Date: {document['date']}")
```

### Basic Web Collection (Document Files)

```python
from codexcollector import CodexCollector

# Configure for web scraping - downloads linked document files
collector = CodexCollector(
    request_delay=2.0,  # 2 seconds between requests
    timeout=30,
    max_crawl_depth=2   # Crawl starting page + 2 levels deep
)

# Collect document files linked on the website
corpus = collector.collect('https://example.com/docs')

# Check for errors
if collector.error_log:
    print(f"Failed to process {len(collector.error_log)} files")
    for source, error in collector.error_log.items():
        print(f"  {source}: {error}")
```

### Web Page Text Extraction

```python
from codexcollector import CodexCollector

# Configure to extract text from web pages themselves
collector = CodexCollector(
    extract_page_text=True,  # Extract HTML page content instead of downloading files
    request_delay=2.0,
    max_crawl_depth=1
)

# Collect text from web pages
corpus = collector.collect('https://example.com/articles')

for page in corpus:
    print(f"Page: {page['filename']}")
    print(f"Content preview: {page['text'][:200]}...")
```

> **Note:** Web page text extraction may encounter challenges such as 404 errors, permission denied responses, or rate limiting from target websites. These are server-side restrictions, not issues with CodexCollector. See the [Web Page Text Extraction Considerations](#web-page-text-extraction-considerations) section for details.

## Configuration

### CodexCollector Parameters

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `supported_extensions` | `set[str]` | `{'.pdf', '.docx', ...}` | File extensions to process (must include leading dot) |
| `max_file_size_mb` | `int` | `100` | Maximum file size in MB (0 = no limit) |
| `request_delay` | `float` | `1.0` | Delay between web requests in seconds |
| `timeout` | `int` | `30` | HTTP request timeout in seconds |
| `excluded_dirs` | `set[str]` | `{'.git', '__pycache__', ...}` | Directory names to skip during traversal |
| `max_crawl_depth` | `int` | `2` | Maximum depth for recursive web crawling (0 = starting page only) |
| `default_encoding` | `str` | `'utf-8'` | Default encoding when chardet fails or as fallback |
| `max_collection_time` | `int` | `0` | Maximum collection time in seconds (0 = no limit) |
| `extract_page_text` | `bool` | `False` | When True, extracts text from HTML pages instead of downloading linked files |

**Default supported extensions:**
```python
{'.pdf', '.docx', '.doc', '.pptx', '.ppt', '.txt', '.md', '.rtf'}
```

**Default excluded directories:**
```python
{'.git', '__pycache__', '.pytest_cache', 'node_modules', '.vscode',
 '.idea', 'venv', '.env', 'logs', 'tmp', 'temp'}
```

## Advanced Usage

### Custom File Filtering

```python
from codexcollector import CodexCollector

# Only collect PDFs and Word documents, limit file size
collector = CodexCollector(
    supported_extensions={'.pdf', '.docx'},
    max_file_size_mb=25,
    excluded_dirs={'archive', 'backup', 'temp'}
)

corpus = collector.collect('/data/documents')
```

### Web Scraping with Depth Control

```python
from codexcollector import CodexCollector

# Configure deep crawling with rate limiting
collector = CodexCollector(
    max_crawl_depth=3,       # Starting page + 3 levels
    request_delay=1.5,       # Respectful crawling
    timeout=60,
    max_collection_time=600  # 10 minute timeout
)

corpus = collector.collect('https://research-papers.example.com')
```

### Progress Logging

```python
from codexcollector import CodexCollector

# Log progress every 100 documents
collector = CodexCollector()
corpus = collector.collect(
    '/large/document/collection',
    progress_callback=100  # Logs at 100, 200, 300, etc. plus final count
)

# Output:
# INFO: Progress: 100 / 500 (20%)
# INFO: Progress: 200 / 500 (40%)
# ...
```

### Custom Text Encoding

```python
from codexcollector import CodexCollector

# Use Latin-1 as fallback for Western European documents
collector = CodexCollector(
    default_encoding='latin-1'
)

corpus = collector.collect('/path/to/european/docs')
```

### Collection with Timeout

```python
from codexcollector import CodexCollector

# Limit collection to 5 minutes
collector = CodexCollector(max_collection_time=300)

corpus = collector.collect('https://large-site.example.com')

print(f"Collected {len(corpus)} documents")
print(f"Errors: {len(collector.error_log)}")
```

## Output Format

### Corpus Structure

The `collect()` method returns a list of document dictionaries:

```python
[
    {
        'filename': 'report.pdf',
        'source': '/path/to/report.pdf',
        'text': 'Document content...',
        'date': '2024-01-15T10:30:00'
    },
    {
        'filename': 'presentation.pptx',
        'source': 'https://example.com/presentation.pptx',
        'text': 'Slide text content...',
        'date': None
    }
]
```

### Document Dictionary Fields

| Field | Type | Description |
|-------|------|-------------|
| `filename` | `str` | Name of the file extracted from path or URL |
| `source` | `str \| None` | Original source path or URL |
| `text` | `str` | Extracted text content (empty string on failure) |
| `date` | `str \| None` | ISO format creation date from metadata, or None if unavailable |

### Type Safety

CodexCollector provides a TypedDict definition for better type hints:

```python
from codexcollector import Document

# Type-safe document creation
doc: Document = {
    'filename': 'example.pdf',
    'source': '/path/to/example.pdf',
    'text': 'Content here',
    'date': '2024-01-01T00:00:00'
}

# Type-safe corpus
corpus: list[Document] = [doc]
```

## Error Handling

CodexCollector implements graceful degradation - individual document failures don't halt collection:

```python
from codexcollector import CodexCollector

collector = CodexCollector()
corpus = collector.collect('/path/to/documents')

# Check error log for failures
if collector.error_log:
    print(f"\nFailed to process {len(collector.error_log)} files:")
    for source, error in collector.error_log.items():
        print(f"  {source}")
        print(f"    Error: {error}")

# Successful documents are still in the corpus
print(f"\nSuccessfully collected {len(corpus)} documents")
```

### Custom Exceptions

```python
from codexcollector import IngestionError, ConfigurationError

try:
    collector = CodexCollector(max_file_size_mb=-1)
except ConfigurationError as e:
    print(f"Invalid configuration: {e}")

# IngestionError is caught internally and logged to error_log
```

## Web Page Text Extraction Considerations

The `extract_page_text=True` option enables extraction of text content directly from HTML pages. While powerful, this feature may encounter challenges depending on target website configurations:

### Common Challenges

| Issue | Cause | Mitigation |
|-------|-------|------------|
| **404 Not Found** | Broken links, dynamic URLs, or removed content | Expected behavior; errors logged to `error_log` |
| **403 Forbidden** | Server blocks automated requests | Increase `request_delay`, respect robots.txt |
| **Rate Limiting** | Too many requests in short period | Use higher `request_delay` (2-5 seconds recommended) |
| **Empty Text** | JavaScript-rendered content | Not supported; use browser automation tools instead |
| **Connection Timeouts** | Slow servers or network issues | Increase `timeout` parameter |

### Best Practices for Web Page Extraction

```python
from codexcollector import CodexCollector

# Conservative configuration for web page extraction
collector = CodexCollector(
    extract_page_text=True,
    request_delay=3.0,       # Generous delay between requests
    timeout=60,              # Allow slow responses
    max_crawl_depth=1,       # Limit scope initially
    max_collection_time=300  # Set overall timeout
)

corpus = collector.collect('https://example.com/articles')

# Always check error_log for issues
print(f"Collected {len(corpus)} pages")
print(f"Errors encountered: {len(collector.error_log)}")

# Review specific errors
for url, error in collector.error_log.items():
    if "403" in error or "Forbidden" in error:
        print(f"Access denied: {url}")
    elif "404" in error:
        print(f"Page not found: {url}")
```

### Content Extraction

When `extract_page_text=True`, CodexCollector:
- Removes boilerplate elements (navigation, headers, footers, sidebars)
- Extracts main content from `<main>`, `<article>`, or similar containers
- Converts HTML to clean text using `html2text`
- Sets `filename` to the URL path
- Sets `date` to None (HTML pages rarely have reliable date metadata)

## Supported File Formats

| Format | Extensions | Library | Metadata |
|--------|------------|---------|----------|
| Word (modern) | `.docx` | `python-docx` | Creation date |
| Word (legacy) | `.doc` | `docx2txt` | None |
| PowerPoint | `.pptx`, `.ppt` | `python-pptx` | Creation date |
| PDF | `.pdf` | `pypdf` | Creation date |
| Plain text | `.txt`, `.md`, `.rtf` | Built-in | None |

**Note:** For files without embedded metadata, the file system modification time is used as a fallback for the `date` field.

## Architecture

### Design Patterns

**Protocol-Based Extension**: Text extractors implement the `TextExtractor` protocol, enabling easy addition of new format support without modifying core code.

```python
class TextExtractor(Protocol):
    def extract_text(self, file_path: Path) -> str: ...
    def extract_metadata(self, file_path: Path) -> dict[str, str | None]: ...
```

**Composition Over Inheritance**: Format-specific extractors are composed into the collector rather than inherited, allowing runtime configuration and testing.

### Key Methods

| Method | Description |
|--------|-------------|
| `collect(source, progress_callback=None)` | Main public API for document collection |
| `_is_url(source)` | Detects whether source is URL or filesystem path |
| `_discover_files_from_path(path)` | Recursive filesystem traversal |
| `_discover_files_from_url(url)` | Web crawling to find document file links |
| `_collect_page_text_from_web(url, ...)` | Web crawling with HTML text extraction |
| `_extract_text_from_html(html, url)` | Extracts main content from HTML pages |
| `_extract_from_file(file_path, source)` | Format-specific text extraction |
| `_download_file(url, temp_path)` | Streaming download for web sources |

## Dependencies

### Core Dependencies

- `beautifulsoup4` - HTML parsing for web scraping
- `chardet` - Character encoding detection
- `docx2txt` - Legacy Word document support
- `html2text` - HTML to text conversion for web page extraction
- `pandas` - Data structure support
- `pypdf` - PDF text extraction
- `python-docx` - Modern Word document parsing
- `python-pptx` - PowerPoint parsing
- `requests` - HTTP requests for web scraping

### Development Dependencies

- `ipykernel` - Jupyter notebook support
- `pytest` - Testing framework

## Testing

Run the included test suite:

```bash
# Using pytest
uv run pytest tests_claude/ -v

# Or directly
pytest tests_claude/ -v
```

## Performance Considerations

### Large Collections

- Use `max_file_size_mb` to skip oversized files
- Set `max_collection_time` to prevent indefinite execution
- Use `progress_callback` with an interval (e.g., `progress_callback=100`) for monitoring

### Web Scraping

- Adjust `request_delay` to respect server rate limits (recommended: >=1.0 seconds)
- Use `max_crawl_depth` to limit scope and prevent excessive crawling
- Set reasonable `timeout` values for slow servers

### Text Encoding

- Use `default_encoding` to skip chardet for known-encoding collections
- For homogeneous sources, setting a specific encoding improves performance

## Limitations

- **PDF extraction**: No OCR support - scanned documents without text layers return empty strings
- **Legacy formats**: `.doc` and `.ppt` files have limited metadata extraction
- **Web scraping**: Respects same-domain restriction; doesn't handle JavaScript-rendered content
- **Web page extraction**: May encounter 404s, 403s, and rate limiting from target servers (see [Web Page Text Extraction Considerations](#web-page-text-extraction-considerations))
- **Encoding detection**: May fail on very short files or files with mixed encodings
- **File formats**: No support for EPUB, ODT, or other formats (extensible via Protocol)

## Use Cases

### Building Text Corpora

```python
from codexcollector import CodexCollector

# Collect academic papers
collector = CodexCollector(supported_extensions={'.pdf'})
corpus = collector.collect('/research/papers')

# Extract just the text for analysis
texts = [doc['text'] for doc in corpus]
```

### Document Archive Mining

```python
from codexcollector import CodexCollector

# Process large archive with progress logging every 100 documents
collector = CodexCollector(
    max_file_size_mb=50,
    excluded_dirs={'backup', 'archive_v1', 'duplicates'}
)

corpus = collector.collect(
    '/corporate/document/archive',
    progress_callback=100
)
```

### Web Documentation Collection

```python
from codexcollector import CodexCollector

# Scrape technical documentation (download linked files)
collector = CodexCollector(
    supported_extensions={'.pdf', '.docx'},
    max_crawl_depth=3,
    request_delay=2.0
)

corpus = collector.collect('https://docs.example.com/api')
```

### Blog/Article Collection

```python
from codexcollector import CodexCollector

# Extract text from blog posts (HTML content)
collector = CodexCollector(
    extract_page_text=True,
    request_delay=2.0,
    max_crawl_depth=2
)

corpus = collector.collect('https://blog.example.com')

# Filter out pages with minimal content
substantial_articles = [doc for doc in corpus if len(doc['text']) > 500]
```

## Contributing

Contributions are welcome! Areas for improvement:

1. **Additional format support**: EPUB, ODT, etc.
2. **OCR integration**: For scanned PDFs
3. **Parallel processing**: For large collections
4. **Incremental collection**: Track processed files to avoid reprocessing
5. **Custom stopword filtering**: At collection time
6. **Robots.txt compliance**: Enhanced web scraping ethics

### Development Process

1. Fork the repository
2. Create a feature branch (`git checkout -b feature/amazing-feature`)
3. Add tests for new functionality
4. Ensure all tests pass (`uv run pytest tests_claude/ -v`)
5. Submit a pull request

## Roadmap

- [ ] OCR integration for scanned documents (pytesseract)
- [ ] Parallel processing for large corpora (multiprocessing)
- [ ] Additional file format support (EPUB, ODT)
- [ ] Robots.txt compliance for web scraping
- [ ] Incremental collection with state persistence
- [ ] Content-type validation before download
- [ ] Language-specific encoding presets
- [ ] Custom extractor registration API
- [ ] JavaScript rendering support (Playwright/Selenium integration)

## Known Issues

- Legacy Word (.doc) and PowerPoint (.ppt) files may not extract text correctly in all cases
- Web crawling doesn't handle JavaScript-rendered content
- Web page text extraction may fail on sites with strict access controls or rate limiting
- Temporary files from web downloads are created in current directory

## License

This project is licensed under the MIT License.

## Support

- Issues: [GitHub Issues](https://github.com/NotAndroid37/codexcollector/issues)

---

Built for seamless document collection and text extraction.