bygod 3.2.0
A comprehensive, truly asynchronous tool for downloading Bible translations from BibleGateway.com
License: MIT
Requires Python: >=3.8
Keywords: bible, download, biblegateway, async, scripture, religion, json, csv, xml, yaml

# ByGoD - The Bible, By God - Bible Gateway Downloader

A comprehensive, truly asynchronous tool for downloading Bible translations from BibleGateway.com in multiple formats (JSON, CSV, YAML, XML) with genuine parallel downloads, retry mechanisms, and flexible output options.

## 🚀 Features

- **True Async HTTP Requests**: Uses `aiohttp` for genuine parallelism, not just threading
- **Direct HTML Parsing**: Bypasses synchronous libraries to directly parse BibleGateway HTML
- **Multiple Translations**: Support for 30+ Bible translations (NIV, KJV, ESV, etc.)
- **Multiple Formats**: Output in JSON, CSV, YAML, and XML formats with consistent structure
- **Format Consistency**: Unified hierarchical organization across all output formats
- **Intelligent Rate Limiting**: Configurable concurrency with automatic rate limiting
- **Retry Mechanisms**: Exponential backoff with configurable retry attempts
- **Organized Output**: Structured directory organization by translation and format
- **Comprehensive Logging**: Colored, detailed progress tracking
- **Flexible Output Modes**: Download individual books, full Bibles, or both

## 📦 Installation

### Option 1: Install from PyPI (Recommended)

```bash
pip install bygod
```

### Option 2: Install from Source (Using Pipenv)

1. **Clone the repository**:
   ```bash
   git clone git@github.com:Christ-Is-The-King/bygod.git
   cd bygod
   ```

2. **Install pipenv** (if not already installed):
   ```bash
   pip install pipenv
   ```

3. **Install dependencies and activate virtual environment**:
   ```bash
   pipenv install
   pipenv shell
   ```

4. **Install in development mode**:
   ```bash
   pip install -e .
   ```

5. **Run the application**:
   ```bash
   python main.py [options]
   ```

### Option 3: Install from Source (Using pip)

1. **Clone the repository**:
   ```bash
   git clone git@github.com:Christ-Is-The-King/bygod.git
   cd bygod
   ```

2. **Create a virtual environment**:
   ```bash
   python -m venv venv
   source venv/bin/activate  # On Windows: venv\Scripts\activate
   ```

3. **Install dependencies**:
   ```bash
   pip install -r requirements.txt
   ```

4. **Install in development mode**:
   ```bash
   pip install -e .
   ```

### Option 4: Build and Install Package

1. **Build the package**:
   ```bash
   python build_package.py
   ```

2. **Install the built package**:
   ```bash
   pip install dist/bygod-*.whl
   ```

## 🎯 Quick Start

### Basic Usage

ByGoD now uses a required positional argument to specify the operation mode:

**Download individual books (all books by default):**
```bash
python main.py books -t NIV -f json
```

**Download specific books only:**
```bash
python main.py books -t NIV -b "Genesis,Exodus,Psalms" -f json
```

**Download entire Bible to a single file:**
```bash
python main.py bible -t NIV -f json
```

**Download both individual books AND entire Bible:**
```bash
python main.py bible-books -t NIV -f json
```

**Download multiple translations in multiple formats:**
```bash
python main.py books -t NIV,KJV,ESV -f json,csv,xml,yaml
```

### Advanced Usage

Download with custom concurrency and retry settings:
```bash
python main.py books \
  -t NIV,KJV \
  -f json,csv \
  -c 10 \
  --retries 5 \
  -d 3 \
  --timeout 600
```

**Operation Modes Explained:**

- **`books`**: Downloads individual book files (all 66 books by default, or use `-b` for specific books)
- **`bible`**: Downloads the entire Bible directly to a single file (most efficient for full Bible only)
- **`bible-books`**: Downloads both individual books AND assembles the full Bible (most comprehensive)

### Verbosity and Logging

Control output verbosity and error logging:

- Use `-v`, `-vv`, or `-vvv` for increasing verbosity
- Use `-q` or `--quiet` to suppress all output except errors
- Use `-e` or `--log-errors` to log errors to a file
- Use `-l` or `--log-level` to set the logging level

**Verbose mode (more detailed output):**

```
bygod -t NIV -m books -v
```

**Log errors to file:**

```
bygod -t NIV --log-errors logs/bible_errors.log
```

**Set specific log level:**

```
bygod -t NIV -ll DEBUG
```

**Combine options:**

```
bygod -t NIV -v --log-errors logs/errors.log -ll WARNING
```

---

## 📋 Sample Log Output

**Books Mode:**
```
12:15:50 - INFO - 🚀 ByGoD
12:15:50 - INFO - 📋 Mode: books
12:15:50 - INFO - 📚 Translations: NIV
12:15:50 - INFO - 📖 Books: All books
12:15:50 - INFO - 📄 Formats: json
12:15:50 - INFO - 📁 Output Directory: ./bibles
12:15:50 - INFO - ⚡ Concurrency: 5 concurrent requests
12:15:50 - INFO - 🔄 Retries: 3 (delay: 2s)
12:15:50 - INFO - ⏱️ Timeout: 300s
12:15:50 - INFO - 📖 Processing NIV
```

**Bible Mode (Assembly):**
```
12:15:50 - INFO - 🚀 ByGoD
12:15:50 - INFO - 📋 Mode: bible
12:15:50 - INFO - 📚 Translations: NIV
12:15:50 - INFO - 📖 Books: All books
12:15:50 - INFO - 📄 Formats: json
12:15:50 - INFO - 📁 Output Directory: ./bibles
12:15:50 - INFO - ⚡ Concurrency: 5 concurrent requests
12:15:50 - INFO - 🔄 Retries: 3 (delay: 2s)
12:15:50 - INFO - ⏱️ Timeout: 300s
12:15:50 - INFO - 📖 Processing NIV
12:15:51 - INFO - 📚 Assembling full Bible for NIV
12:15:51 - INFO - 🔍 Checking for existing book files in ./bibles/NIV/books
12:15:52 - INFO - 🎉 All 66 books found locally - no downloads needed!
12:15:52 - INFO - 💾 Saving full Bible in 1 format(s): json
12:15:53 - INFO - 🎯 Completed full Bible assembly for NIV in 2.45s (reused 66 books, downloaded 0 books)
```

## 📋 Command Line Options

| Option | Description | Default |
|--------|-------------|---------|
| `bygod` | **Required**: Operation mode (books, bible, bible-books) | None |
| `-t, --translations` | Comma-separated list of Bible translations | `NIV` |
| `-b, --books` | Comma-separated list of specific books | All books |
| `-f, --formats` | Output formats: json, csv, xml, yaml | `json` |
| `-o, --output` | Directory to save downloaded Bibles | `./bibles` |
| `--combined` | Generate combined file for multiple translations | `False` |
| `-c, --concurrency` | Maximum concurrent requests | `10` |
| `--retries` | Maximum retry attempts | `3` |
| `-d, --delay` | Delay between retries (seconds) | `2` |
| `--timeout` | Request timeout (seconds) | `300` |
| `-v, --verbose` | Increase verbosity level (-v: INFO, -vv: DEBUG, -vvv: TRACE) | `0` |
| `-q, --quiet` | Suppress all output except errors | `False` |
| `-ll, --log-level` | Set logging level (DEBUG, INFO, WARNING, ERROR, CRITICAL) | `INFO` |
| `--log-errors` | Log errors to specified file | `None` |
| `-dr, --dry-run` | Show what would be downloaded without downloading | `False` |
| `-r, --resume` | Resume interrupted downloads by skipping existing files | `False` |
| `--force` | Force re-download even if files already exist | `False` |

**Operation Modes:**

- **`books`**: Download individual book files (all 66 books by default, or use `-b` for specific books)
- **`bible`**: Download the entire Bible directly to a single file (most efficient for full Bible only)
- **`bible-books`**: Download both individual books AND assemble the full Bible (most comprehensive)

## 📚 Supported Translations

The downloader supports 32 Bible translations:

- **AMP** - Amplified Bible
- **ASV** - American Standard Version
- **AKJV** - Authorized King James Version
- **BRG** - BRG Bible
- **CSB** - Christian Standard Bible
- **EHV** - Evangelical Heritage Version
- **ESV** - English Standard Version
- **ESVUK** - English Standard Version UK
- **GNV** - Geneva Bible
- **GW** - God's Word Translation
- **ISV** - International Standard Version
- **JUB** - Jubilee Bible
- **KJV** - King James Version
- **KJ21** - 21st Century King James Version
- **LEB** - Lexham English Bible
- **LSB** - Legacy Standard Bible
- **MEV** - Modern English Version
- **NASB** - New American Standard Bible
- **NASB1995** - New American Standard Bible 1995
- **NET** - New English Translation
- **NIV** - New International Version
- **NIVUK** - New International Version UK
- **NKJV** - New King James Version
- **NLT** - New Living Translation
- **NLV** - New Life Version
- **NMB** - New Matthew Bible (New Testament only)
- **NOG** - Names of God Bible
- **NRSV** - New Revised Standard Version
- **NRSVUE** - New Revised Standard Version Updated Edition
- **RSV** - Revised Standard Version
- **WEB** - World English Bible
- **YLT** - Young's Literal Translation

## 📁 Output Structure

The downloader creates a well-organized directory structure with consistent formatting across all output formats:

### Format Consistency

All output formats (JSON, YAML, XML, CSV) maintain consistent structure and metadata:

- **Hierarchical Organization**: `language_abbr -> translation_abbr -> book -> chapter -> verse`
- **Language Abbreviations**: 2-character language codes (e.g., "EN" for English, "SP" for Spanish)
- **Metadata Section**: Includes copyright, language, ByGod version, timestamp, and translation info
- **Unified Structure**: Same data organization regardless of output format

### Example Output Structure

```json
{
  "EN": {
    "ESV": {
      "Genesis": {
        "1": {
          "1": "In the beginning, God created the heavens and the earth.",
          "2": "The earth was without form and void...",
          // ... more verses
        }
      }
    }
  },
  "meta": {
    "Copyright": "https://www.biblegateway.com/versions/esv-bible/#copy",
    "Language": "English",
    "ByGod": "3.2.0",
    "Timestamp": "2025-01-XXTXX:XX:XX.XXXXXX+00:00",
    "Translation": "ESV"
  }
}
```

The same hierarchical structure is maintained in YAML, XML, and CSV formats, ensuring data consistency across all outputs.

## ⚡ Performance Optimizations

ByGoD includes several performance optimizations for faster processing:

### Smart Book Reuse
- **Local File Detection**: The `bible_processor` first checks the output directory for existing book files
- **Skip Unnecessary Downloads**: If all 66 books are already present locally, no downloads are performed
- **Efficient Assembly**: Full Bible assembly from local files is significantly faster than re-downloading

### Optimized Bible Assembly
- **Mode Selection**: Choose between `books`, `bible`, or `bible-books` for optimal performance
- **`bible` Mode**: Downloads entire Bible directly (fastest for full Bible only)
- **`bible-books` Mode**: Downloads books first, then assembles (most comprehensive)
- **Parallel Processing**: Multiple books and chapters downloaded concurrently

### Performance Comparison
- **Traditional Approach**: Download all books → Assemble Bible (slower)
- **ByGoD Optimized**: Check local files → Download only missing → Assemble (faster)
- **Typical Speed Improvement**: 2-5x faster when reusing existing book files

### Directory Organization

```
bibles/
├── NIV/
│   ├── bible.json        # Full Bible in JSON
│   ├── bible.csv         # Full Bible in CSV
│   ├── bible.xml         # Full Bible in XML
│   ├── bible.yml         # Full Bible in YAML
│   └── books/
│       ├── Genesis.json  # Individual book in JSON
│       ├── Genesis.csv   # Individual book in CSV
│       ├── Genesis.xml   # Individual book in XML
│       ├── Genesis.yml   # Individual book in YAML
│       └── ...
├── KJV/
│   ├── bible.json
│   ├── bible.csv
│   └── books/
│       └── ...
└── ...
```

## 🏗️ Project Structure

The project has been refactored into a clean, modular structure:

```
bible-gateway-downloader/
├── main.py                    # Main entry point
├── src/                       # Source code package
│   ├── __init__.py
│   ├── constants/             # Bible translations and books data
│   │   ├── translations.py    # BIBLE_TRANSLATIONS dictionary
│   │   ├── books.py          # BOOKS list
│   │   ├── chapters.py       # Chapter counts
│   │   └── cli.py            # CLI constants
│   ├── core/                  # Core downloader functionality
│   │   └── downloader.py      # AsyncBibleDownloader class
│   ├── utils/                 # Utility functions
│   │   ├── formatting.py      # Duration and number formatting
│   │   └── logging.py         # Logging setup and configuration
│   ├── cli/                   # Command line interface
│   │   └── parser.py          # Argument parsing and validation
│   ├── processors/            # Processing logic
│   │   ├── bible.py # Bible download processing
│   │   └── translations.py # Master file processing
│   ├── formatters/            # Output format handlers
│   │   ├── json.py           # JSON formatting
│   │   ├── csv.py            # CSV formatting
│   │   ├── xml.py            # XML formatting
│   │   └── yaml.py           # YAML formatting
│   └── tests/                 # Test suite
│       ├── test_constants.py  # Constants tests
│       ├── test_core.py       # Core functionality tests
│       └── test_utils.py      # Utility tests
├── pyproject.toml             # Project configuration
├── README.md                  # This file
└── ... (other files)
```

## 🔧 Technical Details

### Code Quality Tools

The project includes a comprehensive code quality checking script:

```bash
# Run all quality checks
./scripts/code-checker.sh --all

# Run specific checks
./scripts/code-checker.sh --format    # Black + isort
./scripts/code-checker.sh --lint      # Flake8 + Pylint  
./scripts/code-checker.sh --type      # MyPy type checking
./scripts/code-checker.sh --security  # Bandit + Safety
./scripts/code-checker.sh --docs      # Pydocstyle
./scripts/code-checker.sh --complexity # Vulture + Radon
```

**Current Status**:
- **Formatting**: ✅ All files properly formatted with Black and isort
- **Linting**: ⚠️ Some line length violations remain (mostly long strings/comments)
- **Type Checking**: ⚠️ Type annotations needed in some test files and utility functions
- **Security**: ✅ No critical security issues found
- **Documentation**: ✅ Comprehensive docstrings and README

### True Async Architecture

Unlike traditional threading approaches, this downloader uses:

- **`asyncio`**: Python's native async/await framework
- **`aiohttp`**: True async HTTP client for concurrent requests
- **Semaphores**: Rate limiting with configurable concurrency
- **`asyncio.gather()`**: Parallel execution of multiple downloads

### HTML Parsing

The downloader directly parses BibleGateway HTML using:

- **BeautifulSoup**: HTML parsing and navigation
- **CSS Selectors**: Multiple fallback selectors for verse extraction
- **Regex Patterns**: Chapter discovery and verse number detection

### Modular Architecture

The codebase has been refactored into a clean, modular structure:

- **Separation of Concerns**: Each module has a specific responsibility
- **Maintainability**: Easy to understand and modify individual components
- **Testability**: Each module can be tested independently
- **Reusability**: Core downloader can be imported and used in other projects
- **Code Quality**: Comprehensive linting and formatting standards

### Code Quality Standards

The project maintains high code quality through automated tools:

- **Formatting**: Black for consistent code style, isort for import organization
- **Linting**: Flake8 for style guide enforcement, Pylint for code analysis
- **Type Checking**: MyPy for static type analysis
- **Security**: Bandit for security vulnerability detection, Safety for dependency scanning
- **Documentation**: Pydocstyle for docstring standards
- **Complexity**: Vulture for dead code detection, Radon for complexity analysis

All code is automatically formatted and follows PEP 8 standards.

### Error Handling

- **Exponential Backoff**: Intelligent retry with increasing delays
- **Rate Limit Detection**: Automatic handling of 429 responses
- **Graceful Degradation**: Continues processing even if some downloads fail
- **Detailed Logging**: Comprehensive error reporting and progress tracking

## 🧪 Testing

### Development Environment

The project uses **pipenv** for dependency management:

```bash
# Install dependencies
pipenv install --dev

# Activate virtual environment
pipenv shell

# Run tests
pipenv run pytest src/tests/ -v

# Run code quality checks
pipenv run black src/ main.py
pipenv run isort src/ main.py
pipenv run flake8 src/ main.py
pipenv run mypy src/ main.py
```

### Test Results

Run the test suite to verify functionality:

```bash
# Using pipenv
pipenv run python -m pytest src/tests/ -v

# Run specific test categories
pipenv run python -m pytest src/tests/test_constants.py -v
pipenv run python -m pytest src/tests/test_utils.py -v
pipenv run python -m pytest src/tests/test_core.py -v

# Run with coverage
pipenv run python -m pytest src/tests/ --cov=src --cov-report=html
```

The test suite includes:
- **Core Functionality**: Downloader initialization, context management, request handling
- **Constants Validation**: Bible translations, books, and chapter counts
- **Utilities**: Formatting functions and logging setup
- **Integration Tests**: End-to-end download scenarios

### Test Results
- **47 tests passed** ✅
- **1 test skipped** ⏭️ (complex async mocking)
- **0 tests failed** ❌
- **Clean test suite**: Removed problematic network simulation tests

### Code Quality Status

The project maintains high code quality standards with automated tools:

- **✅ Formatting**: Black (88 char line length) + isort for import organization
- **⚠️ Linting**: Flake8 shows some line length violations (mostly long strings/comments that can't be auto-fixed)
- **⚠️ Type Checking**: MyPy shows type annotation gaps (mostly in test files and some utility functions)
- **✅ Security**: Bandit shows low-risk issues (mostly try-except-pass patterns for cleanup)
- **✅ Import/Export**: Clean import structure with no undefined variables or import errors

**Note**: Some line length violations remain due to long strings, comments, or URLs that cannot be easily reformatted. These are mostly cosmetic and don't affect functionality.

## 📊 Performance

The true async architecture provides significant performance improvements:

- **Genuine Parallelism**: Multiple HTTP requests execute simultaneously
- **Efficient Resource Usage**: No thread overhead, uses event loop
- **Scalable Concurrency**: Configurable rate limits prevent server overload
- **Memory Efficient**: Streams responses without loading entire files into memory

## 🤝 Contributing

1. Fork the repository
2. Create a feature branch
3. Install dependencies using pipenv:
   ```bash
   pipenv install
   pipenv install --dev
   ```
4. Make your changes
5. Add tests for new functionality
6. Ensure all tests pass:
   ```bash
   pipenv run python tests.py
   ```
7. Run the linter to ensure code quality:
   ```bash
   # Run all code quality checks
   ./scripts/code-checker.sh --all
   
   # Or run specific checks
   ./scripts/code-checker.sh --format  # Black + isort
   ./scripts/code-checker.sh --lint    # Flake8 + Pylint
   ./scripts/code-checker.sh --type    # MyPy type checking
   ./scripts/code-checker.sh --security # Bandit + Safety
   ```
8. Submit a pull request

## 📄 License

This project is licensed under the MIT License - see the LICENSE file for details.

## 🙏 Acknowledgments

- BibleGateway.com for providing Bible content
- The Python async community for excellent tools and documentation
- Contributors and users who provide feedback and improvements

## 🆘 Troubleshooting

### Common Issues

**Rate Limiting**: If you encounter 429 errors, reduce the `--concurrency` value.

**Timeout Errors**: Increase the `--timeout` value for slower connections.

**Missing Verses**: Some translations may have different HTML structures. The parser includes multiple fallback methods.

**Memory Usage**: For large downloads, consider downloading fewer books at once or using a lower rate limit.

### Getting Help

- Check the logs for detailed error messages
- Try with a single translation and book first
- Ensure your internet connection is stable
- Verify that BibleGateway.com is accessible from your location