file-parse-by-bajirao 0.1.0
Universal Document Processor for LLM Processing - extracts text, tables, numeric data, and metadata from multiple file formats
Requires Python: >=3.7
Keywords: document processing, pdf, docx, xlsx, csv, ocr, table extraction, llm, text extraction

# Universal Document Processor

Universal Document Processor for LLM Processing - extracts text, tables, numeric data, and metadata from PDF, DOCX, XLSX, CSV, images, HTML, JSON, Markdown, and text files with advanced OCR, intelligent table detection, and numeric extraction.

## Features

- **Multi-format Support**: PDF, DOCX, XLSX, CSV, HTML, JSON, Markdown, images, and text files
- **Advanced OCR**: Multi-engine OCR with automatic fallback (Tesseract, EasyOCR, PaddleOCR)
- **Intelligent Table Detection**: Multiple extraction methods with deduplication
- **Numeric Data Extraction**: Automatic extraction and analysis of numeric metrics
- **LLM-Ready Output**: Formatted context for direct use with LLMs
- **URL Support**: Process documents directly from URLs
- **Comprehensive Metadata**: File type, page count, word count, confidence scores, and more

## Installation

### Basic Installation

```bash
pip install file_parse_by_bajirao
```

### With Optional OCR Engines

```bash
pip install file_parse_by_bajirao[ocr]
```

### With Optional Table Extractors

```bash
pip install file_parse_by_bajirao[tables]
```

### With All Optional Dependencies

```bash
pip install file_parse_by_bajirao[all]
```

### System Dependencies

For OCR functionality, you may need to install Tesseract:

- **Windows**: Download from [GitHub](https://github.com/UB-Mannheim/tesseract/wiki)
- **macOS**: `brew install tesseract`
- **Linux**: `sudo apt-get install tesseract-ocr`

For PDF processing, you may need Poppler:

- **Windows**: Download from [poppler-windows](https://github.com/oschwartz10612/poppler-windows/releases)
- **macOS**: `brew install poppler`
- **Linux**: `sudo apt-get install poppler-utils`

## Quick Start

```python
from file_parse_by_bajirao import UniversalDocumentProcessor

# Initialize processor
processor = UniversalDocumentProcessor()

# Process a document (file path, bytes, or URL)
result = processor.process('document.pdf')

# Get LLM-ready context
llm_context = result.to_llm_context()
print(llm_context)

# Access individual components
print(result.text)
print(result.tables)
print(result.numeric_data)
print(result.metadata)

# Export to JSON
result.to_json("output.json")
```

## Usage Examples

### Process Different File Types

```python
from file_parse_by_bajirao import UniversalDocumentProcessor

processor = UniversalDocumentProcessor()

# PDF file
result = processor.process('document.pdf')

# DOCX file
result = processor.process('document.docx')

# XLSX file
result = processor.process('spreadsheet.xlsx')

# CSV file
result = processor.process('data.csv')

# Image file (with OCR)
result = processor.process('scanned_document.png')

# From URL
result = processor.process('https://example.com/document.pdf')

# From bytes
with open('document.pdf', 'rb') as f:
    result = processor.process(f.read())
```

### Access Extracted Data

```python
# Text content
print(result.text)

# Tables
for table in result.tables:
    print(f"Table {table.table_id} on page {table.page_number}")
    print(table.to_markdown())

# Numeric data
if result.numeric_data:
    print("Key Metrics:", result.numeric_data.key_metrics)
    print("Statistics:", result.numeric_data.statistics)

# Metadata
print(f"File type: {result.metadata.file_type}")
print(f"Pages: {result.metadata.page_count}")
print(f"Words: {result.metadata.word_count}")
print(f"Tables found: {result.metadata.table_count}")
print(f"Confidence: {result.metadata.confidence_score:.0%}")
```

### LLM Context Formatting

```python
# Full context with all features
context = result.to_llm_context(
    include_tables=True,
    include_metadata=True,
    include_numeric=True
)

# Limited context (metadata only)
context = result.to_llm_context(
    include_tables=False,
    include_metadata=True,
    include_numeric=False
)

# With length limit
context = result.to_llm_context(max_length=10000)
```

### Export Results

```python
# Export to JSON file
result.to_json("output.json")

# Get JSON string
json_str = result.to_json()

# Convert to dictionary
data_dict = result.to_dict()
```

## API Reference

### `UniversalDocumentProcessor`

Main processor class.

#### Methods

- `process(source: Union[str, bytes, io.BytesIO]) -> ProcessedDocument`
  - Process a document from file path, bytes, or URL
  - Returns: `ProcessedDocument` object

### `ProcessedDocument`

Result object containing all extracted data.

#### Attributes

- `text: str` - Extracted text content
- `tables: List[TableData]` - Extracted tables
- `numeric_data: Optional[NumericData]` - Numeric data and statistics
- `metadata: DocumentMetadata` - Document metadata
- `errors: List[str]` - Processing errors
- `warnings: List[str]` - Processing warnings

#### Methods

- `to_dict() -> Dict` - Convert to dictionary
- `to_json(filepath: Optional[str] = None) -> str` - Export as JSON
- `to_llm_context(...) -> str` - Format for LLM consumption

### `TableData`

Extracted table structure.

#### Attributes

- `table_id: str` - Unique table identifier
- `page_number: int` - Page number where table was found
- `headers: List[str]` - Table headers
- `rows: List[List[str]]` - Table rows
- `confidence: float` - Extraction confidence (0-1)
- `extraction_method: str` - Method used for extraction

#### Methods

- `to_dict() -> Dict` - Convert to dictionary
- `to_markdown() -> str` - Convert to markdown table

### `NumericData`

Extracted numeric data and statistics.

#### Attributes

- `key_metrics: Dict[str, float]` - Key numeric metrics
- `statistics: Dict[str, Any]` - Statistical summaries
- `numeric_tables: List[pd.DataFrame]` - Numeric-only tables
- `extraction_method: str` - Extraction method used

### `DocumentMetadata`

Document metadata and processing information.

#### Attributes

- `file_type: str` - File type
- `file_size: int` - File size in bytes
- `page_count: int` - Number of pages
- `text_length: int` - Length of extracted text
- `word_count: int` - Word count
- `has_tables: bool` - Whether tables were found
- `table_count: int` - Number of tables found
- `has_numeric: bool` - Whether numeric data was found
- `numeric_count: int` - Number of numeric items
- `extraction_methods: List[str]` - Methods used for extraction
- `processing_time: float` - Processing time in seconds
- `confidence_score: float` - Overall confidence score (0-1)

## Supported File Formats

| Format | Extension | Features |
|--------|-----------|----------|
| PDF | `.pdf` | Text extraction, OCR, table extraction |
| Word | `.docx`, `.doc` | Text and table extraction |
| Excel | `.xlsx` | Sheet and table extraction |
| CSV | `.csv` | Tabular data extraction |
| HTML | `.html` | Text and table extraction |
| JSON | `.json` | Structured data extraction |
| Markdown | `.md` | Text and table extraction |
| Images | `.png`, `.jpg`, `.jpeg`, `.bmp`, `.tiff`, `.gif`, `.webp` | OCR text extraction |
| Text | `.txt` | Plain text extraction |

## Requirements

### Core Dependencies

- PyMuPDF (fitz)
- python-docx
- Pillow
- pytesseract
- pdf2image
- pandas
- numpy
- opencv-python
- openpyxl
- beautifulsoup4
- lxml
- requests

### Optional Dependencies

- **OCR**: easyocr, paddleocr
- **Tables**: pdfplumber, camelot-py, tabula-py

## License

Apache Software License 2.0

## Contributing

Contributions are welcome! Please feel free to submit a Pull Request.

## Author

Bajirao S. Sali

## Support

For issues, questions, or contributions, please visit the [GitHub repository](https://github.com/bajirao/universal-document-processor).