scholar-flux 0.4.0
The ScholarFlux API is an open-source project designed to streamline access to academic and scholarly resources across various platforms for discovery and analyses. It offers a unified API that simplifies querying academic databases, retrieving metadata, and performing comprehensive searches within scholarly articles, journals, and publications.
License: Apache-2.0
Requires Python: <4,>=3.10

![ScholarFluxBanner](assets/Banner.png)

[![codecov](https://codecov.io/gh/sammieh21/scholar-flux/graph/badge.svg?token=D06ZSHP5GF)](https://codecov.io/gh/sammieh21/scholar-flux)
[![CI](https://github.com/SammieH21/scholar-flux/actions/workflows/ci.yml/badge.svg)](https://github.com/SammieH21/scholar-flux/actions/workflows/ci.yml)
[![CodeQL](https://github.com/SammieH21/scholar-flux/actions/workflows/github-code-scanning/codeql/badge.svg)](https://github.com/SammieH21/scholar-flux/actions/workflows/github-code-scanning/codeql)
[![Documentation Status](https://github.com/SammieH21/scholar-flux/actions/workflows/docs.yml/badge.svg)](https://github.com/SammieH21/scholar-flux/actions/workflows/docs.yml)
[![Python 3.10+](https://img.shields.io/badge/python-3.10+-blue.svg)](https://www.python.org/downloads/)
[![License](https://img.shields.io/badge/License-Apache_2.0-blue.svg)](https://opensource.org/licenses/Apache-2.0)

![PyPI - Version](https://img.shields.io/pypi/v/scholar-flux)
[![Beta](https://img.shields.io/badge/status-beta-yellow.svg)](https://github.com/SammieH21/scholar-flux)
[![mypy: Type Checked](https://www.mypy-lang.org/static/mypy_badge.svg)](https://mypy-lang.org/)
[![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)
[![Linting: Ruff](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/charliermarsh/ruff/main/assets/badge/v2.json)](https://github.com/astral-sh/ruff)



## Table of Contents

- [Overview](#overview)
- [Installation](#-installation)
- [Quick Start](#-quick-start)
- [Origin Story](#origin-story)
- [Architecture](#architecture)
- [Core Features](#core-features)
- [Supported Providers](#supported-providers)
- [Comparison with Existing Tools](#comparison-with-existing-tools)
- [What's New in v0.4.0](#whats-new-in-v040)
- [Documentation](#-documentation)
- [Contributing](#contributing)
- [License](#license)
- [Acknowledgments](#acknowledgments)
- [Contact](#contact)

**Quick Links:**
- **Home**: https://github.com/SammieH21/scholar-flux
- **Documentation**: https://SammieH21.github.io/scholar-flux/
- **Source Code**: https://github.com/SammieH21/scholar-flux/tree/main/src/scholar_flux
- **Contributing**: https://github.com/SammieH21/scholar-flux/blob/main/CONTRIBUTING.md
- **Code Of Conduct**: https://github.com/SammieH21/scholar-flux/blob/main/CODE_OF_CONDUCT.md
- **Issues**: https://github.com/SammieH21/scholar-flux/issues
- **Security**: https://github.com/SammieH21/scholar-flux/blob/main/SECURITY.md


## Overview

ScholarFlux is production-grade orchestration infrastructure for academic APIs, initially developed during a **4-year CDC public health analysis fellowship** to address the challenges researchers face when aggregating data across multiple scholarly databases. It enables **concurrent multi-provider search with automatic rate limiting, streaming result delivery, and intelligent schema normalization** across 7+ scholarly databases—arXiv, PubMed, Springer Nature, Crossref, OpenAlex, PLOS, and CORE.

Query multiple academic databases simultaneously while ScholarFlux handles provider-specific quirks, rate limits, response validation, and format inconsistencies—delivering ML-ready datasets with consistent schemas.

### The Problem

Academic research requires querying multiple databases, but each provider implements their own parameter names, pagination mechanisms, rate limits, error conditions, and response formats. Building integrations with multiple academic APIs typically means:

- Manually coordinating rate limits across providers (6s for PLOS, 10s per batch request for CORE, 3s for arXiv...)
- Writing custom parsers for XML (PubMed, arXiv) and JSON (Crossref, OpenAlex) responses
- Mapping inconsistent field names and data types across separate APIs and databases
- Implementing retry logic that handles connection errors, internal server errors, client-side errors
- Building caching layers to avoid redundant requests
- Handling provider-specific pagination quirks and knowing when to stop requesting
- Managing API tokens and sensitive fields securely and efficiently
- Handling provider-specific workflows required to retrieve records effectively (PubMed)

**Result**: Weeks of integration work just to retrieve data consistently.

### The Solution

ScholarFlux handles and abstracts away the complexity of retrieving and processing data from Academic APIs through provider-specific rate limiting, concurrent thread orchestration, streaming result delivery, automatic schema normalization, two-tier caching with production backends (Redis, MongoDB, SQLAlchemy, DuckDB), and security-first credential masking. The result: **~3x faster** multi-provider retrieval with consistent, ML-ready output. The architecture handles rate limiting coordination, response validation, and intelligent retry logic automatically.


### Who Is This For?

- **Researchers** conducting systematic literature reviews and meta-analyses across databases
- **Data Engineers** building academic pipelines requiring reliable retrieval, caching, request throttling with robust error handling
- **ML Practitioners** requiring consistently structured, ML-ready datasets with preprocessed fields

### Features at a Glance

- **Security-first** - Identifies and masks sensitive data (API keys, emails, credentials) before they appear in logs
- **Rate limiting** - Automatically respects per-provider rate limits to avoid getting banned
- **Request preparation** - Configures provider-specific API parameters and settings for data retrieval
- **Intelligent halting** - After unsuccessful requests, knows when to retry or halt multi-page retrieval
- **Response validation** - Verifies response structure before attempting to process data
- **Concurrent orchestration** - Retrieves data from multiple APIs concurrently with multithreading
- **Record processing** - Prepares, logs, and returns intermediate data steps and final processed results
- **API-Aware Normalization** – Consolidates API-specific record structures into a unified, ML/analytics-ready schema
- **Two-layer caching** - Optionally caches successful requests and response processing to avoid redundant requests

## Focus

- **Unified Access**: Aggregate searches across multiple academic databases and publishers
- **Rich Metadata Retrieval**: Fetch detailed metadata for each publication, including authors, publication date, abstracts, and more
- **Advanced Search Capabilities**: Support both simple searches and provider-specific, complex query structures to filter by publication date, authorship, and keywords
- **Open Access Integration**: Prioritize and query open-access resources (for use within the terms of service for each provider)
- **Production-Ready Architecture**: Built with dependency injection, comprehensive error handling, and type safety for deployment in production environments

## 📦 Installation

### Prerequisites

- Python 3.10+
- [Poetry](https://python-poetry.org/) for dependency management (for development)
- An API key depending on the API Service Provider (may be available through your academic institution or by registering directly with the provider)

### Provider Access

While some APIs may require an API key, the majority of providers do not. **OpenAlex, PLOS, Crossref, CORE, and arXiv work out-of-the-box** and seamlessly for both single-page and multi-page/provider retrieval, even with the default settings.

APIs such as Springer Nature, while requiring API keys, provide API access without payment or subscription for uses within the terms of service. PubMed and the CORE API, while not requiring API keys, greatly increase the allowable requests per second with an API key.

All sources have rate limits that users should abide by to prevent `Too Many Requests` status codes. ScholarFlux handles rate limiting automatically.

### Basic Installation

```bash
pip install scholar-flux
```

This installs the core package with minimal dependencies for JSON-based providers (PLOS, OpenAlex, Crossref).

### Installation with Extras

```bash
# For XML parsing (PubMed, arXiv workflows)
pip install scholar-flux[parsing]

# For production caching (Redis, MongoDB, SQLAlchemy)
pip install scholar-flux[database]

# For DuckDB support (embedded analytical database)
pip install scholar-flux[duckdb]

# For encrypted session caching
pip install scholar-flux[cryptography]
```

### Configuration (Optional)

ScholarFlux works out of the box, but environment variables enable higher rate limits and cleaner deployments:

```bash
# Identify your application (recommended for API providers)
export SCHOLAR_FLUX_DEFAULT_USER_AGENT="MyApp/1.0 (mailto:you@institution.edu)"

# Enable "polite pool" access for Crossref/OpenAlex (10x higher rate limits)
export SCHOLAR_FLUX_DEFAULT_MAILTO=your.email@institution.edu

# Centralize config, cache, and logs (recommended for Docker/production)
export SCHOLAR_FLUX_HOME=/opt/scholar-flux

# Default cache backends (optional - memory/sqlite used otherwise)
export SCHOLAR_FLUX_DEFAULT_SESSION_CACHE_BACKEND=redis      # HTTP response cache
export SCHOLAR_FLUX_DEFAULT_RESPONSE_CACHE_STORAGE=redis     # Processed result cache
```

If the User-Agent is not set, the `requests` library will use its default when sending requests (e.g., `python-requests/2.32.5`). While the default user-agent may be accepted, rate limits may be lower for certain APIs as a result.

For production deployments with Redis/MongoDB, API keys, and Docker configuration, see the [Production Deployment Guide](https://SammieH21.github.io/scholar-flux/production_deployment.html).

## 🚀 Quick Start

### Simplest Example

Just want to see it work? Here's the absolute minimum:

```python
from scholar_flux import SearchCoordinator

coordinator = SearchCoordinator(query="machine learning", provider_name="arxiv")
result = coordinator.search_page(1)

if result:
    print(f"Success! Got {result.record_count} records")

    # Note: result.data might be empty if the query matches no documents
    if result.data:
        print(f"Title of first article: {result.data[0].get('title')}")
    else:
        print("Query returned no results. Try a different search term.")
else:
    print(f"Error: {result.error}")
```

### Complete Example

For real-world usage with normalization and error handling:

```python
from scholar_flux import SearchCoordinator
from scholar_flux.utils import JsonFileUtils
from datetime import date
from pathlib import Path

# Create a coordinator for a single provider
coordinator = SearchCoordinator(query="machine learning", provider_name="arxiv")

# Search and get results:
page = 1
response = coordinator.search_page(page=page)

print(response)
# OUTPUT: SearchResult(query='machine learning', provider_name='arxiv', page=1, response_result=ProcessedResponse(...))

if response:
    # show the total number of records that were retrieved and processed
    print(f"Found {response.total_query_hits} total results for the query, {response.query}")
    print(f"Retrieved {response.record_count} records from page {response.page}")

    # Access processed data with predictable fields:
    normalized_records = response.normalize()
    for article in normalized_records:
        abstract = article.get('abstract')
        summary = abstract[:80] + '...' if abstract else 'Not Found'

        print(f"Title: {article.get('title')}")
        print(f"Authors: {article.get('authors')}")
        print(f"Abstract: {summary}")

    # Optionally write the result to a JSON file to your documents folder (Note: arXiv is fundamentally open access)
    filename = Path.home() / "Documents" / f"arXiv_ml_page_{page}-{date.today()}.json"
    JsonFileUtils.save_as(normalized_records, filename)
    print(f"Records written to '{filename}'!")
else:
    print(f"Oops, An error occurred during response retrieval for page {response.page}: ", response.error, response.message)
```

## Origin Story

Initially developed during a 4-year CDC Public Health Analyst Fellowship as an exploratory project investigating how AI and ML could enhance research workflows. The challenge: aggregating data from multiple academic databases for ML-driven research, where each provider has different APIs, rate limits, and response formats.

Early prototypes of these AI/ML-driven workflows revealed that reliable data integration was critical—without consistent, validated data from heterogeneous sources, downstream analysis fails.

Built and presented at CDC meetings as a solution for AI-assisted systematic literature review and meta-analysis workflows. The initial demonstration showcased a Springer Nature integration with embedding-based similarity search to find related articles and abstracts—illustrating how unified API access could power ML-driven research discovery.

After the fellowship, I recognized the broader need beyond public health research and open-sourced it, expanding from the initial Springer Nature integration to 7+ providers with comprehensive documentation and production-ready features.

**Technical foundation:**
- **~58,319 lines of code**: ~32,777 LOC source + ~25,542 LOC comprehensive tests
- **97% test coverage**: Rigorous testing across all functionality and edge cases
- **Security-focused**: Automated CVE scanning, credential masking, encrypted caching
- **Type-safe**: comprehensive mypy type checking all throughout the entire codebase
- **Production-ready architecture**: Dependency injection, comprehensive error handling, horizontal scaling

## Architecture

ScholarFlux is built around three core components that work together through dependency injection:

```
SearchCoordinator
├── SearchAPI (HTTP retrieval + rate limiting)
│   ├── RateLimiter (thread-safe rate limiting with Retry-After support)
│   ├── Session (requests or requests-cache)
│   ├── APIParameterMap (provider-specific parameter translation)
│   ├── SensitiveDataMasker (masks sensitive data before logging)
│   └── SearchAPIConfig (records per page, request delays, provider URL/name, API keys)
│
├─── ResponseCoordinator (processing pipeline)
│   ├── DataParser (XML/JSON/YAML → dict)
│   ├── DataExtractor (dict → records list)
│   ├── DataProcessor (records transformation with filtering)
│   ├── ResponseMetadataMap (pagination metadata extraction - v0.3.0)
│   └── DataCacheManager (In-Memory, Redis, MongoDB, SQLAlchemy, or DuckDB Storage Cache Devices)
├────── RetryHandler (exponential backoff with configurable limits)
├────── ResponseValidator (Defines the logic used to verify context integrity)
└────── SearchWorkflow (Optional provider-specific workflow for multi-step, paginated searches)
```

### Concurrency Architecture

For multi-provider searches, ScholarFlux uses a sophisticated threading model with shared rate limiters:

```
MultiSearchCoordinator
├── Thread Pool (per-provider threads)
│   ├── Thread 1: PLOS (shared rate limiter across all PLOS queries)
│   │   └── Concurrent: query1_page1, query1_page2, query1_page3 → (waits 6s between)
│   ├── Thread 2: arXiv (shared rate limiter)
│   ├── Thread 3: OpenAlex (shared rate limiter)
│   └── Thread 4: Crossref (shared rate limiter)
│
├── Shared Rate Limiter Registry (cross-query coordination)
└── Generator Pipeline (streaming results via concurrent.futures.as_completed)
```

**Key Design Decisions**:
- **Threading over asyncio**: Simpler for users, better for I/O-bound workloads with rate limits
- **Generator-based streaming**: Memory-efficient, process results incrementally without blocking
- **Shared rate limiters**: Multiple queries to the same provider coordinate through a single `ThreadedRateLimiter`
- **Concurrent execution**: Maximizes throughput by requesting from all providers simultaneously within rate limit constraints

Each component has a specific responsibility:

- **SearchAPI**: Creates HTTP requests and handles provider-specific parameter building
- **ResponseCoordinator**: Orchestrates parsing → extraction → transformation → caching
- **SearchCoordinator**: Delegates between SearchAPI (retrieval) and ResponseCoordinator (processing)

Supporting components include:

- **SensitiveDataMasker**: Pattern matching to identify, mask, and register sensitive strings (API keys, tokens)
- **DataParser**: Parses XML, JSON, and YAML responses into dictionaries
- **DataExtractor**: Extracts records from nested dictionaries with configurable paths
- **DataProcessor**: Transforms records using field mappings and filtering rules
- **ResponseMetadataMap** *(v0.3.0)*: Extracts pagination metadata across provider-specific field names
- **DataCacheManager**: Manages caching backends (In-Memory, Redis, MongoDB, SQLAlchemy, DuckDB)
- **RateLimiter**: Enforces per-provider rate limits with proactive `Retry-After` detection *(v0.3.0)*
- **RetryHandler**: Implements exponential backoff with case-insensitive header parsing *(v0.3.0)*

### How Concurrent Orchestration Works

**Minimal Example**: Retrieving 200 records across 3 providers — 2 pages each

```python
from scholar_flux import SearchCoordinator, MultiSearchCoordinator
# ❌ Sequential approach (traditional)
query = "machine learning"
pages = [1, 2]
plos = SearchCoordinator(query=query, provider_name='plos')
crossref = SearchCoordinator(query=query, provider_name='crossref')
arxiv = SearchCoordinator(query=query, provider_name='arxiv') # requires `xmltodict`

results_plos = plos.search_pages(pages)  # Request → waits 6 seconds between requests
results_arxiv = arxiv.search_pages(pages)  # Request → waits 4 seconds between requests
results_crossref = crossref.search_pages(pages)  # Request → waits 1 second between requests

# Total: ~12-13 seconds for 6 requests (the delay between requests plus processing time adds up)

# ✅ ScholarFlux concurrent threading (default)
multi_search_coordinator = MultiSearchCoordinator.from_coordinators([plos, arxiv, crossref])
results = multi_search_coordinator.search_pages(pages=pages)  # multithreading=True by default

# What happens:
# t=0s: All threads request the first page simultaneously
# t=~0.5s: All responses received → rate limiters activate
# t=6-7s: PLOS completes (slowest, determines total time)
# Total: ~6-7 seconds (bottlenecked by slowest provider)
# Speedup: Approximately 2x faster than sequential (~6s vs ~12s)
```

### Performance: Real-World Benchmarks

This optimization compounds with multiple pages and providers:

**Scenario**: Retrieve ~1,650 records across 6 providers (Crossref, PLOS, arXiv, OpenAlex, Springer Nature, PubMed) — 10 pages each

| Method                           | Time       | Speedup          |
|----------------------------------|------------|------------------|
| Sequential requests              | ~2.5 min   | Baseline         |
| ScholarFlux concurrent threading | ~45 sec    | **~3x faster**   |

**Why the speedup?** While one provider waits on rate limits, others continue simultaneously. With varied delays across providers (1s–6s), concurrent execution maximizes throughput.

> **Note on CORE API**: CORE is fully supported but enforces stricter burst limits beyond the documented 10s/request delay. After ~10–13 consecutive requests, you may encounter 429 errors and multi-minute cooldowns. For large CORE queries, consider reducing records per request or spreading retrieval across multiple sessions.



### Multi-Provider Search with Normalization

```python
from scholar_flux import SearchCoordinator, MultiSearchCoordinator

# Create coordinators for multiple providers (None of the following require an API key)
providers = ['crossref', 'arxiv', 'openalex', 'plos', 'core']
# Set your own user agent to help APIs identify the origin of the request:
user_agent = None # 'MyResearchProject/1.0 (mailto:your.email@institution.edu)'
coordinators = [
    SearchCoordinator(query="CRISPR gene editing", provider_name=provider, user_agent=user_agent)
    for provider in providers
]

# Coordinate concurrent searches
multi_search_coordinator = MultiSearchCoordinator.from_coordinators(coordinators)

# Use `search_page` to retrieve a single page across several providers
results = multi_search_coordinator.search_page(1)

# Filter successful results and normalize to a common schema, including the formatted provider name
normalized_results = results.filter().normalize(include={'display_name'})
for record in normalized_results[:5]:
    print(f"Title: {record['title']} ({record['year']})")
    print(f"Source: {record['display_name']} ({record['url']})")
    print("-"*100)
```


## Core Features

### Rate Limiting

ScholarFlux respects per-provider rate limits automatically:

```python
coordinator = SearchCoordinator(query="sleep", provider_name='plos')

# Each request waits as needed to maintain the rate limit
results = coordinator.search_pages(pages=range(1, 3))
```

**Default Rate Limits:**

ScholarFlux implements conservative rate limits that respect each provider's requirements:

- **PLOS**: 6.1 seconds between requests
- **arXiv**: 4 seconds between requests
- **OpenAlex**: 1 second between requests (conservative—OpenAlex uses 5 metrics)
- **PubMed**: 2 seconds between requests (3 req/sec → 10 req/sec with API key)
- **Crossref**: 1 second between requests
- **CORE API**: 10 seconds between requests (token-based, not request-based)
- **Springer Nature**: 2 seconds between requests

**Override the default delay:**
```python
# Override the default delay for a provider
coordinator = SearchCoordinator(query = 'AI in Academia', provider_name = "arXiv", request_delay=5)
# Or set the delay dynamically during a search
response = coordinator.search(page=1, request_delay=3.1)
```

**Note:** Retry attempts and dynamic throttling are handled at the level of the `RetryHandler` for successive failed requests. If a provider requests a `Retry-After` delay longer than the configured `max_backoff` (default: 120s), ScholarFlux will raise a `RetryAfterDelayExceededException` to prevent indefinite waiting. When this occurs during coordinated searches, an `ErrorResponse` is returned, containing the complete response details for inspection and debugging. You can adjust this behavior by increasing the `max_backoff` or disabling strict enforcement with `RetryHandler.RAISE_ON_DELAY_EXCEEDED = False`. See the [Response Handling Patterns Tutorial](https://SammieH21.github.io/scholar-flux/response_handling_patterns.html) for details.

### Two-Tier Caching

ScholarFlux implements two caching layers:

1. **HTTP Response Caching** (Layer 1): Uses `requests-cache` to cache raw API responses
2. **Processed Result Caching** (Layer 2): Caches extracted and processed records

```python
from scholar_flux import SearchCoordinator, CachedSessionManager, DataCacheManager

# HTTP response caching
session_manager = CachedSessionManager(backend='sqlite', expire_after=3600)
session = session_manager()
processing_cache = DataCacheManager.with_storage('memory')  # or sql/sqlite if SQLAlchemy is available

# Both layers working together
coordinator = SearchCoordinator(
    query="neuroscience",
    provider_name='pubmed',
    session=session,  # Layer 1: HTTP caching
    cache_manager=processing_cache # Layer 2: Response processing cache
)

response = coordinator.search(page=1)  # Fetches from API
response2 = coordinator.search(page=1)  # Instant return from cache
```

For production deployments with Redis or MongoDB, see the [Caching Strategies Tutorial](https://SammieH21.github.io/scholar-flux/caching_strategies.html).

**Note:** ScholarFlux supports in-memory and persistent caching for raw responses and response processing cache for efficiency and rate limiting.
Always review the terms of service for each provider before caching or storing retrieved data.


### API-Specific Parameters

Specific APIs often define custom logic for sorting, filtering, and other provider-native options that can be used for custom searches:

```python
# OpenAlex: Sorting by citation count and filtering by year
coordinator = SearchCoordinator(query="sunspots", provider_name="openalex")
results = coordinator.search_pages(range(1, 5), sort="cited_by_count:desc", filter="publication_year:2024")

# Crossref: Polite pool access for 10x rate limits:
coordinator = SearchCoordinator(query="neural networks", provider_name="crossref")
results = coordinator.search_pages(range(1, 3), sort="published", order="desc", mailto="you@institution.edu")

# Use `.describe()` to see all accepted universal and a minimal, extensible set of API-specific parameters for a provider:
print(coordinator.api.describe())

```

### Schema Normalization

ScholarFlux normalizes provider-specific field names into a common academic schema:

```python
# Raw records have provider-specific field names
results = coordinator.search_pages(pages=range(1, 5))

# Normalize to universal schema
normalized = results.normalize()

# Now all records have consistent fields:
# 'title', 'doi', 'authors', 'abstract', 'journal', 'year', etc.
df = pd.DataFrame(normalized)
```

For custom field mappings and advanced normalization, see the [Schema Normalization Tutorial](https://SammieH21.github.io/scholar-flux/schema_normalization.html).

### Workflow Automation

Some providers (like PubMed) require multiple API calls. ScholarFlux handles this automatically:

```python
# This single call executes a two-step workflow automatically:
# 1. PubMedSearch: Get article IDs
# 2. PubMedFetch: Retrieve full articles with abstracts
coordinator = SearchCoordinator(query="neuroscience", provider_name='pubmed')
result = coordinator.search(page=1)

# Complete metadata preserved across workflow steps
print(result.metadata)  # Query info, ID lists, result counts
print(result.data)      # Full article records with abstracts
```

See the [Advanced Workflows Tutorial](https://SammieH21.github.io/scholar-flux/advanced_workflows.html) for custom multi-step workflows.

### Non-Paginated Endpoint Support

Use `parameter_search()` to query specialized endpoints that don't require pagination (recommendations, citations, metadata lookups, full text retrieval):

```python
result = coordinator.parameter_search(
    endpoint="/articles/recommend",  # mock endpoint
    article_id="PMC1234567",  # mock ID parameter
    limit=20  # mock page-limit parameter
)
```

This method is used internally by workflows for multi-step retrieval patterns. See [Advanced Workflows](https://SammieH21.github.io/scholar-flux/advanced_workflows.html) for actual implementations in workflows.

### Response Validation & Error Handling

ScholarFlux validates responses at multiple stages, gracefully handling and reporting errors when they occur. When timeouts, missing API keys, missing dependencies, and other errors occur, the `SearchCoordinator` gracefully handles and returns error messages via response classes to indicate what went wrong, all the while ensuring that retrieval and processing pipelines across multiple providers aren't brought to a halt.

The client provides three distinct response types:

```python
response = coordinator.search(page=1)

if response:  # Falsy if error or no response
    # ProcessedResponse - successful retrieval and processing
    print(f"Retrieved {len(response.data)} records")
    print(f"Total available: {response.total_query_hits}")
else:
    # ErrorResponse or NonResponse - something went wrong
    print(f"Error: {response.message}")
    print(f"Error type: {response.error}")
```

**Response types:**
- `ProcessedResponse`: Successful retrieval and processing
- `ErrorResponse`: Retrieved response but encountered processing error
- `NonResponse`: Failed to retrieve response (connection error, timeout, etc.)


## Supported Providers

ScholarFlux includes pre-configured support for these academic databases:

| Provider            | Search | Normalization | Special Features                   |
| ------------------- | ------ | ------------- | ---------------------------------- |
| **arXiv**           | ✅     | ✅            | Preprints, categories              |
| **Crossref**        | ✅     | ✅            | DOI metadata, funding              |
| **CORE**            | ✅     | ✅            | Open access aggregator             |
| **OpenAlex**        | ✅     | ✅            | Comprehensive metadata             |
| **PLOS**            | ✅     | ✅            | Open access biology                |
| **PubMed**          | ✅     | ✅            | Two-step workflow (search → fetch) |
| **Springer Nature** | ✅     | ✅            | Requires API key                   |

All providers support:
- Automatic rate limiting with proactive `Retry-After` handling
- Two-tier caching (HTTP + processed results)
- Intelligent pagination with metadata extraction
- Schema normalization with fallback field paths
- Comprehensive error handling and retries

For adding custom providers, see the [Custom Provider Tutorial](https://SammieH21.github.io/scholar-flux/custom_providers.html).


## Comparison with Existing Tools

ScholarFlux is **not a replacement** for single-provider clients like `habanero`, `pybliometrics`, or `arxiv`. It's an **orchestration layer** that complements these tools for multi-provider research workflows.

### Architectural Differences

**Existing packages** (`habanero`, `pybliometrics`, `arxiv`, `metapub`, `scholarly`):
- Single-provider API wrappers
- Provider-specific response structures
- Basic or no caching
- Sequential request patterns
- Designed for provider-specific features

**ScholarFlux**:
- Multi-provider orchestration engine
- Unified schema normalization across providers
- Two-tier caching (HTTP + processed results)
- Concurrent threading with shared rate limiters
- Production-ready architecture (Redis, MongoDB, SQLAlchemy)

### Feature Comparison

| Feature | ScholarFlux | habanero | pybliometrics | arxiv | metapub |
|---------|-------------|----------|---------------|-------|---------|
| **Multi-provider concurrent execution** | ✅ | ❌ | ❌ | ❌ | ❌ |
| **Shared rate limiter coordination** | ✅ | ❌ | ⚠️ Single provider | ❌ | ⚠️ Single provider |
| **Two-tier caching system** | ✅ | ❌ | ⚠️ Basic file cache | ❌ | ❌ |
| **Cross-provider schema normalization** | ✅ | ❌ | ❌ | ❌ | ❌ |
| **Metadata-driven pagination** | ✅ | ❌ | ❌ | ❌ | ❌ |
| **Proactive rate limiting** (`Retry-After`) | ✅ | ❌ | ❌ | ❌ | ❌ |
| **Resilient field mapping** (fallback paths) | ✅ | ❌ | ❌ | ❌ | ❌ |
| **Streaming generator results** | ✅ | ❌ | ❌ | ❌ | ❌ |
| **Multi-step workflow automation** | ✅ | ❌ | ❌ | ❌ | ⚠️ PubMed only |
| **Production cache backends** | ✅ Redis, MongoDB, SQL | ❌ | ⚠️ File system | ❌ | ❌ |
| **Security features** (credential masking) | ✅ | ❌ | ❌ | ❌ | ❌ |
| **Type safety** (mypy checked) | ✅ | ⚠️ Partial | ⚠️ Partial | ⚠️ Partial | ❌ |

### What ScholarFlux Adds

1. **Concurrent Orchestration**: Query 7+ providers simultaneously—**3x faster** than sequential
2. **Metadata-Driven Intelligence**: Extract pagination metadata for precise control
3. **Resilient Schema Normalization**: Normalize **120 provider-specific fields** with fallback paths
4. **Proactive Rate Limiting**: Prevent 429 errors by reading `Retry-After` headers
5. **Production Infrastructure**: Redis/MongoDB/SQL caching, credential masking, comprehensive error handling
6. **Workflow Automation**: Multi-step APIs handled transparently with metadata preservation
7. **Memory Efficiency**: Stream results as they arrive—process page 1 while fetching page 100

### When to Use Each Approach

**Use provider-specific packages** (`habanero`, `arxiv`, `pybliometrics`) when:
- You need **one database** with provider-specific advanced features
- You want **fine-grained control** over provider-specific parameters
- You're building **provider-specific workflows** not covered by ScholarFlux

**Use ScholarFlux** when:
- You need **3+ databases** queried concurrently
- You need **consistent schemas** for ML/analytics pipelines
- You need **production reliability** with comprehensive error handling
- You need **metadata-driven pagination** that knows when to stop
- You need **resilient field mapping** that handles API inconsistencies
- You're building **production systems** requiring caching and horizontal scaling
- You want **rapid prototyping** without orchestration boilerplate


### Real-World Scenario

**Without ScholarFlux** (using individual packages):
```python
# Researcher needs data from 4 sources
from habanero import Crossref
import arxiv
from pymed import PubMed

# Manual threading implementation
# Manual rate limiting for each provider
# Manual schema normalization across 120 field variations
# Manual caching layer for both requests and results
# Manual error handling with retry logic
# Manual workflow orchestration for PubMed's two-step process
# Result: 200+ lines of boilerplate code
```

**With ScholarFlux**:
```python
from scholar_flux import SearchCoordinator, MultiSearchCoordinator
import pandas as pd

# Each coordinator automatically handles parameter mappings and URL structures under the hood:
coordinators = [
    SearchCoordinator(query="CRISPR", provider_name=provider_name, use_cache=True)
    for provider_name in ("crossref", "arxiv", "pubmed", "plos")
]

# Automatic threaded execution with rate limiting
multi_search_coordinator = MultiSearchCoordinator.from_coordinators(coordinators)

# Search pages 1-10 across all providers simultaneously
results = multi_search_coordinator.search_pages(pages=range(1, 11))

# Identify the total number of records retrieved per provider:
for coordinator in multi_search_coordinator.coordinators:
    print(
        f"Total record count for {coordinator.display_name}:",
        results.select(provider_name=coordinator.provider_name).record_count,
    )

# Metadata-driven pagination intelligence
for result in results:
    print(f"{result.display_name} Page {result.page}: {result.record_count} records ({result.total_query_hits:,} hits)")

# Automatic normalization with API-specific post processing. The `include` field adds search-specific fields to the output:
df = pd.DataFrame(
    results.filter().normalize(keep_api_specific_fields=False, include={"query", "page", "provider_name"})
)

# Output columns after record normalization:
print(df.columns)
# ['provider_name', 'doi', 'url', 'record_id', 'title', 'abstract', 'authors', 'journal', 'publisher', 'year', 'date_published',
#  'date_created', 'keywords', 'subjects', 'full_text', 'citation_count', 'open_access', 'license', 'record_type', 'language',
#  'is_retracted', 'query', 'page']

# Result: 14 lines, production-ready
```


**Complementary use**: ScholarFlux can be extended to wrap existing packages for providers it doesn't support natively. See the [Custom Provider Tutorial](https://SammieH21.github.io/scholar-flux/custom_providers.html).

For detailed comparison, see the [documentation](https://SammieH21.github.io/scholar-flux/).

## What's New in v0.4.0

v0.4.0 delivers **API-Aware Normalization Post-Processing Pipelines**, **Request Observability Infrastructure**, and **Production Hardening** for data engineering, research, and ML/AI use cases.

### API-Aware Post-Processing Pipelines

ScholarFlux now includes intelligent post-processing pipelines for each provider, transforming raw API responses into consistent, ML-ready records.

Previously, normalization was tasked with mapping API-specific fields to universal field names, retrieving and extracting elements from configured fallback paths when possible.
Now each provider implements a `_post_process()` method that transforms processed, API-specific records into consistent, validated output.

| Transformation | Raw Value | Normalized Output |
|----------------|-----------|-------------------|
| Year extraction (`PubMed`) | `[[2024, 6, 15]]` | `2024` |
| Author formatting (`Crossref`) | `[{"given": "Jane", "family": "Smith"}]` | `["Jane Smith"]` |
| OA resolution (`Crossref`) | `"creativecommons.org/licenses/by/4.0/"` | `true` |
| Abstract reconstruction (`OpenAlex`) | `{"Hello": [0], "world": [1]}` | `"Hello world"` |

The post-processing step for the current provider is applied automatically when calling `normalize()`:

```python
from scholar_flux import SearchCoordinator
from scholar_flux.utils import truncate

# Query Crossref—raw responses have nested, provider-specific structures
coordinator = SearchCoordinator(query="CRISPR gene editing", provider_name="OpenAlex")
response = coordinator.search_page(1)

# Normalize with automatic post-processing:
for article in response.normalize():
    print(f"Title: {article['title']}")
    print(f"Authors: {article['authors']}")       # ["Jennifer Doudna", "Feng Zhang"]
    print(f"Year: {article['year']}")              # 2024 (extracted from date-parts)
    print(f"Open Access: {article['open_access']}") # True (resolved from license URL)
    print(f"Abstract: {truncate(article['abstract'] or 'N/A', max_length = 120)}") # Reconstructed Abstract
    print("*" * 120)
```

**What normalization handles automatically:**
- **Nested field traversal**: Paths like `MedlineCitation.Article.AuthorList.Author` (PubMed) or `authorships.institutions.display_name` (OpenAlex)
- **Fallback paths**: Variability in record fields is handled by using fallback paths for different record types when the default data location is empty

**Examples of what Post-processing additionally provides:**
- **Type conversions**: Dates to ISO format, years as integers, booleans from strings
- **Open access detection**: License URL pattern matching (Crossref), PMCID presence (PubMed), default OA status (CORE)
- **Author formatting**: Crossref and pubmed fields maps parse nested lists of author dictionaries containing first, middle, and last names as separate fields into a normalized list of author names (e.g., `given`/`family` for Crossref and `ForeName`/`Initials`/`LastName` for PubMed).
- **Abstract Reconstruction**: The `OpenAlexFieldMap` automatically parses abstract inverted indexes to reconstruct abstracts into human-readable formats. The `CrossrefFieldMap` extracts and removes HTML tags to clean abstract texts for downstream applications.
- **URL reconstruction**: PLOS and PubMed article URLs built from DOI/PMID identifiers
- **Cross-database identifiers**: CORE extracts arXiv ID, PMID, and MAG ID for entity resolution

### Record Resolution for ML Pipelines

When building ML pipelines, you often need to trace processed records back to their original nested structure. ScholarFlux now optionally annotates records with content-based hashes for bidirectional record linking. This happens internally during record normalization to reliable retrieve fields consistently across record types:

```python
from scholar_flux import SearchCoordinator

# Query Crossref—raw responses have nested, provider-specific structures
coordinator = SearchCoordinator(query="Genomics breakthroughs", provider_name="crossref", annotate_records=True)
response = coordinator.search_page(1)

# Processed records may be flattened or filtered
processed_record = response.processed_records[0]

# These two fields are added by default if `annotate_records=True`
print(f"The hash of record {processed_record['_extraction_index']}: is {processed_record['_record_id']}")
# OUTPUT: The hash of record 0: is 29ffdeea3452d26b_0

# Resolve back to the original extracted structure:
original_record = response.resolve_extracted_record(0)  # Or processed_record['_extraction_index']

# Clean records for export (removes internal annotations):
clean_records = response.strip_annotations()
```

### Request History and Observability

Debug rate limiting and retry b