semantic-qa-gen 0.1.0
A Python library for generating high-quality question-answer pairs from PDF, DOCX, MD, and TXT files
License: MIT
Requires Python: >=3.10
Keywords: ai, fine-tuning, llm, nlp, qa-pairs, question-answering, question-generation, semantic-analysis

<div align="center">
    <h1>SemanticQAGen</h1>
  <p><em>Intelligent Question-Answer Generation with Advanced Semantic Understanding For AI Fine-Tuning</em></p>
</div>

<p align="center">
  <a href="#overview">Overview</a> •
  <a href="#installation">Installation</a> •
  <a href="#quickstart">Quickstart</a> •
  <a href="#feature-overview">Feature Overview</a> •
  <a href="#core-capabilities">Core Capabilities</a> •
  <a href="#advanced-features">Advanced Features</a> •
  <a href="#project-file-organization">Project File Organization</a> •
  <a href="#architecture">Architecture</a> •
  <a href="#configuration">Configuration</a> •
  <a href="#api-reference">API Reference</a> •
  <a href="#cli-reference">CLI Reference</a> •
  <a href="#usage-examples">Usage Examples</a> •
  <a href="#extension">Extension</a> •
  <a href="#troubleshooting">Troubleshooting</a> •
  <a href="#license">License</a>
</p>

---
> **Alpha Release (v0.10)**: This library is in active development and not functional yet. I estimate it is 90% complete, so the last %10 percent may take 90% of the time. However, the feature set is amazing IMO, and I am looking forward to completing this library. If you are interested in helping, please fork the repo and help bring this Python library to life. 
---

## Overview

SemanticQAGen is a powerful Python library for generating high-quality question-answer pairs from text documents. It uses advanced semantic understanding to intelligently process content, analyze information density, and create diverse questions across multiple cognitive levels.

This builds upon concepts from the [Augmentoolkit](https://github.com/e-p-armstrong/augmentoolkit) project by Evan Armstrong and other QA generation projects. I tried Evan's project on some test data and fell in love with it. I originally intended to fork his project and work on adding features I wanted to see. After making the plan, I realized that this wasn't simply a fork. It just felt wrong to try and completely redesign someone else's excellent library and I wanted a much different feature set and so this library is the result. 

SemanticQAGen features enhanced semantic chunking, dynamic question generation, optional *validation* of questions and answers using retrieval-augmented generation (RAG), and flexible LLM routing capabilities. You can run all tasks locally on an OpenAI compatible server, run them off an Internet hosted API, or split specific tasks (eg. chunking, validation, analysis, generation) between local and remote servers. The library is designed with the "for Humans" philosophy - simple for basic use cases while providing advanced capabilities for power users.

---

## Installation

### Basic Installation

```bash
pip install semantic-qa-gen
```

### With Optional Dependencies

```bash
# With PDF support
pip install semantic-qa-gen[pdf]

# With full document support (PDF, DOCX, etc.)
pip install semantic-qa-gen[docs]

# With RAG validation capabilities
pip install semantic-qa-gen[rag]

# With enhanced UI and NLP capabilities
pip install semantic-qa-gen[full]

# Development installation with testing tools
pip install semantic-qa-gen[dev]
```

### Requirements

- Python 3.10 or higher
- Required dependencies are automatically installed with the package

---

## Quickstart

### Basic Usage

```python
from semantic_qa_gen import SemanticQAGen

# Initialize with default settings
qa_gen = SemanticQAGen()

# Process a document
result = qa_gen.process_document("path/to/document.txt")

# Save the questions to a JSON file
qa_gen.save_questions(result, "output")
```

### CLI Usage

```bash
# Generate questions from a document with default settings
semantic-qa-gen process document.pdf -o questions_output

# Process a whole directory of documents
semantic-qa-gen process-batch input_directory/ -o output_directory/

# Create a config file interactively 
semantic-qa-gen init-config config.yml --interactive

# Process with a specific configuration
semantic-qa-gen process document.txt --config config.yml --format json
```

---

## Feature Overview

SemanticQAGen offers a comprehensive set of features designed to produce high-quality question and answer sets:

| Feature Category | Capability                                                         | Included |
|-----------------|--------------------------------------------------------------------|:--------:|
| **Document Processing** | Document format support: TXT, PDF, DOCX, MD                      | ✅ |
|  | Batch document processing                                          | ✅ |
|  | Automatic document type detection                                  | ✅ |
|  | Cross-page content handling                                        | ✅ |
|  | Header/footer detection and removal                                | ✅ |
| **Content Analysis** | Semantic document chunking                                         | ✅ |
|  | Information density analysis                                       | ✅ |
|  | Topic coherence evaluation                                         | ✅ |
|  | Key concept extraction                                             | ✅ |
|  | Educational level classification                                   | ✅ |
| **Question Generation** | Multi-level cognitive questions (factual, inferential, conceptual) | ✅ |
|  | Adaptive generation based on content quality                       | ✅ |
|  | Question diversity enforcement                                     | ✅ |
|  | Custom question categories                                         | ✅ |
| **Answer Validation** | Factual accuracy verification                                      | ✅ |
|  | RAG-enhanced fact checking                                         | ✅ |
|  | Question clarity evaluation                                        | ✅ |
|  | Answer completeness assessment                                     | ✅ |
| **LLM Integration** | OpenAI API support                                                 | ✅ |
|  | Local LLM support (Ollama, etc.)                                   | ✅ |
|  | Hybrid task routing                                                | ✅ |
|  | Automatic fallback mechanisms                                      | ✅ |
| **Processing Control** | Checkpoint and resume capability                                   | ✅ |
|  | Parallel processing                                                | ✅ |
|  | Progress tracking and reporting                                    | ✅ |
|  | Memory optimization                                                | ✅ |
| **Output Options** | Multiple export formats (JSON, CSV)                                | ✅ |
|  | Metadata inclusion                                                 | ✅ |
|  | Statistics and analytics                                           | ✅ |
|  | RAG retriever creation                                             | ✅ |
| **Extensibility** | Custom document loaders                                            | ✅ |
|  | Custom chunking strategies                                         | ✅ |
|  | Custom validators                                                  | ✅ |
|  | Plugin architecture                                                | ✅ |

---

## Core Capabilities

### Document Processing

#### Multiple Format Support
SemanticQAGen can read and process a variety of document formats including plain text, PDF, Markdown, and DOCX. Each format is handled by specialized loaders that extract content while preserving document structure.

```python
# Process different file types the same way
result_txt = qa_gen.process_document("document.txt")
result_pdf = qa_gen.process_document("document.pdf")
result_md = qa_gen.process_document("document.md")
result_docx = qa_gen.process_document("document.docx")
```

#### Batch Processing
Process multiple files in a single operation. The system can handle directories of mixed document types and will generate separate outputs for each.

```python
# Process all files in a directory
qa_gen.process_batch(
    input_dir="input/",
    output_dir="output/",
    file_types=["txt", "pdf", "docx", "md"]
)
```

#### Automatic Document Type Detection
The system automatically detects document types using both file extensions and content analysis, ensuring the correct loader is used even when file extensions are missing or incorrect.

#### Cross-Page Content Handling
For PDF documents, the system intelligently handles sentences and paragraphs that span across page boundaries, creating a seamless text flow for better semantic analysis.

#### Header/Footer Detection
Automatic detection and optional removal of repeating headers and footers in PDF documents, preventing them from being included in generated questions.

### Content Analysis

#### Semantic Document Chunking
Documents are intelligently broken down into semantically coherent chunks based on content structure rather than arbitrary size limits. This preserves context and produces more meaningful question-answer pairs.

```python
# Configure chunking strategy
config = {
    "chunking": {
        "strategy": "semantic",  # Options: semantic, fixed_size, hybrid
        "target_chunk_size": 1500,
        "preserve_headings": True
    }
}
```

#### Information Density Analysis
Each chunk is analyzed for information density - how rich in facts and teachable content it is. This analysis guides question generation to focus on content-rich sections.

#### Topic Coherence Evaluation
The system evaluates how well each chunk maintains a coherent topic or theme, which helps ensure generated questions relate to a consistent subject area.

#### Key Concept Extraction
Important concepts, terms, and ideas are automatically identified in each chunk, forming the basis for targeted question generation.

#### Educational Level Classification (Optional)
Content is classified by appropriate educational level (elementary, high school, undergraduate, etc.) to help generate questions at suitable complexity levels. You can remove this metadata if it isn't applicable to your dataset.

### Question Generation

#### Multi-level Cognitive Questions
The system generates questions across three cognitive domains:
- **Factual**: Direct recall of information stated in the content
- **Inferential**: Questions requiring connecting multiple pieces of information
- **Conceptual**: Higher-order questions about principles, implications, or broader understanding

```python
# Configure question categories
config = {
    "question_generation": {
        "categories": {
            "factual": {"min_questions": 3, "weight": 1.0},
            "inferential": {"min_questions": 2, "weight": 1.2},
            "conceptual": {"min_questions": 1, "weight": 1.5}
        }
    }
}
```

#### Adaptive Generation
The number and types of questions generated adapt automatically based on content quality. Information-dense chunks yield more questions, while sparse chunks yield fewer.

#### Question Diversity Enforcement
To avoid repetitive or overly similar questions, the system enforces diversity by comparing newly generated questions with existing ones and filtering out duplicates.

#### Custom Question Categories
Users can define custom question categories beyond the standard factual/inferential/conceptual to target specific learning objectives.

### Answer Validation

#### Factual Accuracy Verification
All generated answers are verified against the source content to ensure they do not contain factual errors or hallucinations.

#### RAG-Enhanced Fact Checking
Optional LlamaIndex-based retrieval-augmented validation ensures answers are firmly grounded in the source material. This provides an additional verification layer beyond prompt-based checking.

#### Question Clarity Evaluation
Questions are evaluated for clarity and unambiguity, filtering out poorly formed questions that might confuse learners.

#### Answer Completeness Assessment
The system checks that answers thoroughly address the questions asked, eliminating partial or incomplete responses.

---

## Advanced Features

### RAG-Enhanced Answer Validation

SemanticQAGen's most advanced validation feature uses Retrieval-Augmented Generation (RAG) through LlamaIndex to ensure factual correctness. Unlike many AI question generators that can "hallucinate" incorrect facts, the RAG validation system acts as a powerful factual grounding mechanism.

#### How RAG Validation Works

1. **Document Indexing**: Source documents are indexed using LlamaIndex's document-centric architecture
2. **Answer Verification**: Generated answers are validated against the source material using specialized evaluation metrics
3. **Factual Grounding**: The CorrectnessEvaluator compares answers directly with retrieved context to detect discrepancies
4. **Confidence Scoring**: Each answer receives a factual grounding score (0.0-1.0) indicating reliability

#### Why I Chose LlamaIndex Over Alternatives

I personally prefer LangChain architecture, however LlamaIndex was selected specifically for its document-centric approach, specialized evaluation tools, and embedding management capabilities, making it ideal for factual verification. This is strictly a validation mechanism - it does not pollute or alter the Q/A generation process. The source material remains the single source of truth.

```python
# Enable RAG validation
config = {
    "validation": {
        "factual_accuracy": {
            "enabled": True,
            "threshold": 0.7
        },
        "rag_factual": {
            "enabled": True,
            "model": "gpt-4",
            "threshold": 0.75
        },
        "use_enhanced_validation": True  # Combine traditional and RAG validation
    }
}
```

#### RAG Validator Benefits

- **Source-Grounded Answers**: Ensures all information in answers comes directly from the source material
- **Hallucination Prevention**: Drastically reduces the likelihood of generated answers containing false information
- **Enhanced Quality Control**: Provides a robust verification layer beyond simple prompt engineering

The RAG validation system can be configured to be more or less strict depending on your needs. When using the enhanced combined validation, it benefits from both traditional prompt-based and retrieval-based approaches.

### LLM Integration

#### OpenAI API Support
Full integration with OpenAI with optimized prompting strategies for each task in the pipeline.

#### Local LLM Support
Support for local LLM deployment via Ollama, LM Studio, and similar services, allowing use of models like Llama, Mistral, etc., without requiring external API access.

#### Hybrid Task Routing
Intelligently route different tasks to the most appropriate LLM based on task complexity and model capability. For example, use GPT-4 for complex question generation but a local model for simple validation tasks.

```python
config = {
    "llm_services": {
        "local": {
            "enabled": True,
            "url": "http://localhost:11434/api",
            "model": "mistral:7b",
            "default_for": ["chunking", "validation"]
        },
        "remote": {
            "enabled": True,
            "provider": "openai",
            "model": "gpt-4",
            "default_for": ["analysis", "generation"]
        }
    }
}
```

#### Automatic Fallback Mechanisms
If a primary LLM service fails, the system automatically tries fallback services, ensuring robustness in production environments.

### Processing Control

#### Checkpoint and Resume Capability
Processing can be interrupted and resumed later using a checkpoint system. This is essential for large documents or when processing must be paused.

```python
config = {
    "processing": {
        "enable_checkpoints": True,
        "checkpoint_dir": "./checkpoints",
        "checkpoint_interval": 10  # Save every 10 chunks
    }
}
```

#### Parallel Processing
Multi-threaded processing of documents and chunks with configurable concurrency levels to maximize throughput on multi-core systems.

#### Progress Tracking and Reporting
Detailed progress reporting during processing, with support for both simple console output and rich interactive displays.

#### Memory Optimization
Smart memory management techniques to handle very large documents without exhausting system resources.

### Output Options

#### Multiple Export Formats
Export question-answer pairs in various formats including JSON, CSV, with customizable formatting options.

```python
# Save questions in different formats
qa_gen.save_questions(result, "questions_output", format_name="json")
qa_gen.save_questions(result, "questions_output", format_name="csv")
```

#### Metadata Inclusion
Include rich metadata about source documents, generation parameters, and validation results with the generated questions.

#### Statistics and Analytics
Comprehensive statistics about generated questions, including category distribution, validation success rates, and content coverage.

#### RAG Retriever Creation
Generated questions and answers can be exported as a LlamaIndex retriever for use in downstream applications.

```python
# Create a retriever from generated QA pairs
retriever = qa_gen.create_qa_retriever(result, api_key="YOUR_OPENAI_API_KEY")
response = retriever.retrieve("How does photosynthesis work?")
```

---

## Project File Organization
```txt
.
├── 📄 __init__.py
├── 📄 semantic_qa_gen.py
├── 📄 version.py
├── 📁 semantic_qa_gen
│   ├── 📄 __init__.py
│   ├── 📄 main.py
│   ├── 📄 semantic_qa_gen.py
│   ├── 📁 chunking
│   │   ├── 📄 analyzer.py
│   │   ├── 📄 engine.py
│   │   └── 📁 strategies
│   │       ├── 📄 base.py
│   │       ├── 📄 fixed_size.py
│   │       ├── 📄 nlp_helpers.py
│   │       └── 📄 semantic.py
│   ├── 📁 cli
│   │   ├── 📄 __init__.py
│   │   └── 📄 commands.py
│   ├── 📁 config
│   │   ├── 📄 manager.py
│   │   └── 📄 schema.py
│   ├── 📁 document
│   │   ├── 📄 models.py
│   │   ├── 📄 processor.py
│   │   └── 📁 loaders
│   │       ├── 📄 base.py
│   │       ├── 📄 docx.py
│   │       ├── 📄 markdown.py
│   │       ├── 📄 pdf.py
│   │       └── 📄 text.py
│   ├── 📁 llm
│   │   ├── 📄 router.py
│   │   ├── 📄 service.py
│   │   ├── 📁 adapters
│   │   │   ├── 📄 local.py
│   │   │   └── 📄 remote.py
│   │   └── 📁 prompts
│   │       └── 📄 manager.py
│   ├── 📁 output
│   │   ├── 📄 __init__.py
│   │   ├── 📄 formatter.py
│   │   └── 📁 adapters
│   │       ├── 📄 __init__.py
│   │       ├── 📄 csv.py
│   │       └── 📄 json.py
│   ├── 📁 pipeline
│   │   └── 📄 semantic.py
│   ├── 📁 question
│   │   ├── 📄 generator.py
│   │   ├── 📄 processor.py
│   │   └── 📁 validation
│   │       ├── 📄 base.py
│   │       ├── 📄 diversity.py
│   │       ├── 📄 engine.py
│   │       └── 📄 factual.py
│   └── 📁 utils
│       ├── 📄 checkpoint.py
│       ├── 📄 error.py
│       ├── 📄 logging.py
│       └── 📄 progress.py
├── 📄 setup.py
└── 📁 tests
    ├── 📄 conftest.py
    ├── 📄 test_chunking.py
    ├── 📄 test_config.py
    ├── 📄 test_document_processor.py
    ├── 📄 test_llm_service.py
    ├── 📄 test_main.py
    ├── 📄 test_pipeline.py
    ├── 📄 test_question_generation.py
    ├── 📄 test_utils.py
    ├── 📄 test_validation.py
    └── 📁 integration
        └── 📄 test_minimal_pipeline.py
```

## Architecture

SemanticQAGen implements a modular pipeline architecture with clearly defined components and interfaces:

```
                                   ARCHITECTURE
┌───────────────────────────────────────────────────────────────────────────────┐
│                                                                               │
│                       ┌─────────────────────────────────┐                     │
│                       │       SemanticQAGen Class       │                     │
│                       │      (Main User Interface)      │                     │
│                       └─────────────┬───────────────────┘                     │
│                                     │                                         │
│                                     ▼                                         │
│           ┌────────────────────────────────────────────────────┐              │
│           │              SemanticPipeline Orchestrator         │              │
│           └┬────────────────┬───────────────────┬─────────────┬┘              │
│            │                │                   │             │               │
│  ┌─────────▼──────────┐     │    ┌──────────────▼───────────┐ │               │
│  │  Document Manager  │     │    │   Chunking & Analysis    │ │               │
│  └┬────────────────┬──┘     │    └┬─────────────────────┬───┘ │               │
│   │                │        │     │                     │     │               │
│┌──▼───────┐   ┌────▼────┐   │  ┌──▼───────┐       ┌────▼────┐ │               │
││ Document │   │Document │   │  │ Chunking │       │Semantic │ │               │
││ Loaders  │   │Processor│   │  │ Engine   │       │Analyzer │ │               │
│└──────────┘   └─────────┘   │  └──────────┘       └─────────┘ │               │
│                             │                                 │               │
│      ┌─────────────────────────────────────────────────┐      │               │
│      │               LLM Service Router                │      │               │
│      │                                                 │      │               │
│      │  ┌────────────────┐         ┌────────────────┐  │      │               │
│      │  │ Remote LLM     │         │ Local LLM      │  │      │               │
│      │  │ (OpenAI, etc.) │         │ (Ollama, etc.) │  │      │               │
│      │  └────────────────┘         └────────────────┘  │      │               │
│      └─────────────────────────────────────────────────┘      │               │
│                             │                                 │               │
│  ┌────────────────────────┐ │ ┌───────────────────────────────▼────────┐      │
│  │  Question Generator    │ │ │         Validation Engine              │      │
│  │                        │◄┼─┼────┐                                   │      │
│  │  ┌──────────────────┐  │ │ │    │                                   │      │
│  │  │Category: Factual │  │ │ │    │  ┌─────────────┐ ┌──────────────┐ │      │
│  │  └──────────────────┘  │ │ │    ├─►│ Traditional │ │  RAG-based   │ │      │
│  │  ┌──────────────────┐  │ │ │    │  │ Validators  │ │  Validator   │ │      │
│  │  │Cat: Inferential  │  │ │ │    │  └─────────────┘ └──────────────┘ │      │
│  │  └──────────────────┘  │ │ │    │                                   │      │
│  │  ┌──────────────────┐  │ │ │    │  ┌──────────────────────────────┐ │      │
│  │  │Cat: Conceptual   │  │ │ │    └─►│  Enhanced Validator          │ │      │
│  │  └──────────────────┘  │ │ │       └──────────────────────────────┘ │      │
│  └─────────┬──────────────┘ │ └───────────────────────────────────────┬┘      │
│            │                │                                         │       │
│            └────────────────┼─────────────────────────────────────────┘       │
│                             │                                                 │
│          ┌──────────────────▼─────────────────────┐                           │
│          │           Output Formatter             │                           │
│          │                                        │                           │
│          │  ┌─────────────┐    ┌────────────────┐ │                           │
│          │  │ JSON Adapter│    │  CSV Adapter   │ │                           │
│          │  └─────────────┘    └────────────────┘ │                           │
│          └────────────────────────────────────────┘                           │
│                             │                                                 │
│           ┌─────────────────▼────────────────────┐                            │
│           │           Output Results             │                            │
│           │  • Questions & Answers               │                            │
│           │  • Document Metadata                 │                            │
│           │  • Statistics                        │                            │
│           └──────────────────────────────────────┘                            │
│                                                                               │
│                      ┌─────────────────────────────┐                          │
│                      │     Checkpoint Manager      │                          │
│                      │   (Resume Capabilities)     │                          │
│                      └─────────────────────────────┘                          │
│                                                                               │
│                      ┌─────────────────────────────┐                          │
│                      │     Progress Reporter       │                          │
│                      │   (Processing Feedback)     │                          │
│                      └─────────────────────────────┘                          │
└───────────────────────────────────────────────────────────────────────────────┘

                                 DATA FLOW
┌───────────────────────────────────────────────────────────────────────────────┐
│                                                                               │
│  ┌───────────┐      ┌───────────┐     ┌───────────┐      ┌───────────┐        │
│  │  Document │      │ Document  │     │   List    │      │Chunk List │        │
│  │  Files    │─────►│   Model   │────►│ of Chunks │─────►│w/Analysis │        │
│  └───────────┘      └───────────┘     └───────────┘      └───────────┘        │
│                                                                │              │
│                                                                ▼              │
│                                         ┌────────────────────────────────┐    │
│  ┌───────────┐     ┌────────────┐       │ Question-Answer Generation     │    │
│  │   Final   │     │  Validated │       │ ┌────────────┐ ┌────────────┐  │    │
│  │  Output   │◄────┤ Question   │◄──────┤ │ Question 1 │ │ Question 2 │  │    │
│  │           │     │  List      │       │ └────────────┘ └────────────┘  │    │
│  └───────────┘     └────────────┘       └────────────────────────────────┘    │
│                                                                               │
└───────────────────────────────────────────────────────────────────────────────┘

                           VALIDATION SUBSYSTEM
┌───────────────────────────────────────────────────────────────────────────────┐
│                                                                               │
│   ┌───────────────┐     ┌─────────────────────────────────────────────────┐   │
│   │ Generated     │     │             Validation Engine                   │   │
│   │ Questions     │────►│                                                 │   │
│   └───────────────┘     │    ┌────────────────────┐  ┌─────────────────┐  │   │
│                         │    │  Traditional Flow  │  │   RAG Flow      │  │   │
│                         │    │                    │  │                 │  │   │
│                         │    │ ┌──────────────┐   │  │ ┌─────────────┐ │  │   │
│                         │    │ │  Factual     │   │  │ │ Source      │ │  │   │
│                         │    │ │  Accuracy    │   │  │ │ Document    │ │  │   │
│                         │    │ └──────────────┘   │  │ │ Index       │ │  │   │
│                         │    │ ┌──────────────┐   │  │ └──────┬──────┘ │  │   │
│                         │    │ │  Answer      │   │  │        │        │  │   │
│                         │    │ │ Completeness │   │  │ ┌──────▼──────┐ │  │   │
│                         │    │ └──────────────┘   │  │ │ LlamaIndex  │ │  │   │
│                         │    │ ┌──────────────┐   │  │ │ Correctness │ │  │   │
│                         │    │ │  Question    │   │  │ │ Evaluator   │ │  │   │
│                         │    │ │  Clarity     │   │  │ └──────┬──────┘ │  │   │
│                         │    │ └──────────────┘   │  │        │        │  │   │
│                         │    │ ┌──────────────┐   │  │ ┌──────▼──────┐ │  │   │
│                         │    │ │  Diversity   │   │  │ │ Accuracy    │ │  │   │
│                         │    │ │  Check       │   │  │ │ Score       │ │  │   │
│                         │    │ └──────────────┘   │  │ └─────────────┘ │  │   │
│                         │    └────────────────────┘  └─────────────────┘  │   │
│                         │                                                 │   │
│                         │            ┌────────────────────┐               │   │
│                         │            │   Combine Results  │               │   │
│                         │            └─────────┬──────────┘               │   │
│                         │                      │                          │   │
│                         └──────────────────────┼──────────────────────────┘   │
│                                                │                              │
│   ┌───────────────┐                  ┌─────────▼──────────┐                   │
│   │ Final Valid   │◄─────────────────┤ Validation Results │                   │
│   │ Questions     │                  └────────────────────┘                   │
│   └───────────────┘                                                           │
│                                                                               │
└───────────────────────────────────────────────────────────────────────────────┘

                             BATCH PROCESSING 
┌───────────────────────────────────────────────────────────────────────────────┐
│                                                                               │
│   ┌─────────────┐         ┌───────────────────────┐      ┌──────────────┐     │
│   │ Input       │         │ Batch Processor       │      │ Checkpoint   │     │
│   │ Directory   │─────────►                       │◄────►│ Store        │     │
│   └─────────────┘         │ ┌────────────────┐    │      └──────────────┘     │
│                           │ │ File Type      │    │                           │
│                           │ │ Detection      │    │                           │
│                           │ └────────────────┘    │                           │
│                           │ ┌────────────────┐    │                           │
│                           │ │ Processing     │    │                           │
│                           │ │ Queue          │    │                           │
│                           │ └────────────────┘    │                           │
│                           │ ┌────────────────┐    │      ┌──────────────┐     │
│                           │ │ Parallel       ├────────────► File 1      │     │
│                           │ │ Scheduler      │    │      │ Processing   │     │
│                           │ └──┬────────────┬┘    │      └──────────────┘     │
│                           │    │            │     │      ┌──────────────┐     │
│                           │    │            ├────────────► File 2       │     │
│                           │    │            │     │