aip-engine 0.4.1
Algebraic Independence Processor — auto-detection of matrix structure + memory-efficient computation for ultra-large sparse systems
License: MIT
Requires Python: >=3.8
Keywords: sparse-matrix, linear-algebra, memory-efficient, combinatorial-number-system, accordion-memory, large-scale-computation, scientific-computing

# AIP Engine

**Algebraic Independence Processor** — auto-detection of matrix structure + memory-efficient computation for ultra-large sparse systems.

```
pip install aip-engine
```

## What it does

AIP Engine solves a real problem: building and solving sparse linear systems that are **too large for conventional tools**. It does three things:

1. **Detects** matrix structure automatically (sparse/dense, square/rectangular)
2. **Routes** to the optimal solver (LSQR, spsolve, LAPACK)
3. **Accordion Memory**: builds and solves ultra-large systems without running out of RAM

## Quick start

```python
import aip

# Auto-detect structure and solve
report = aip.detect_matrix(A)
x = aip.solve(A, b)
```

## Accordion Memory

For systems with millions or billions of entries that don't fit in RAM:

```python
from aip.accordion import PascalIndex, AccordionBuilder, solve_chunks

# 1. Mathematical indexing (replaces dictionary, saves GB of RAM)
index = PascalIndex(num_vars=30, max_degree=10)
idx = index.combo_to_index((3, 7, 12))  # O(k) time, 0 extra memory

# 2. Batch construction (never all in RAM at once)
builder = AccordionBuilder(num_rows=53_000_000)
# ... add entries in batches ...
builder.flush()  # converts to CSR chunk, frees raw data
chunks = builder.finalize()

# 3. Streaming solve (never assembles full matrix)
result = solve_chunks(chunks, b, max_iter=10000)
print(result['residual'], result['size_l2'])
```

## Why Accordion?

| | Without Accordion | With Accordion |
|---|---|---|
| Monomial index (8.6M entries) | ~2 GB dictionary | 0 MB (computed mathematically) |
| Matrix construction (150M entries) | ~12 GB Python lists | ~2.4 GB array.array per batch |
| Full matrix (53M x 1.17B) | 496,052 TB dense | 14.5 GB sparse chunks |
| Solve | needs full matrix in RAM | streaming over chunks |

Real-world results:

| Problem | Matrix size | Dense would be | Accordion uses | Compression |
|---|---|---|---|---|
| PHP n=4 d=8 | 8.6M x 78M | 5.4 PB | 1.2 GB | 4,640,586x |
| PHP n=5 d=10 | 53M x 1.17B | 496,052 TB | 14.5 GB | 34,215,310x |

## How it works

### PascalIndex

Uses the [Combinatorial Number System](https://en.wikipedia.org/wiki/Combinatorial_number_system) to compute the index of any monomial in O(k) time using a precomputed Pascal table. No dictionary needed.

```python
index = PascalIndex(num_vars=45, max_degree=10)
print(index)  # PascalIndex(vars=45, deg=10, monomials=4,346,814,276, pascal=4048 bytes)
# 4.3 billion monomials indexed with 4 KB of memory
```

### AccordionBuilder

Builds sparse matrices in batches using `array.array` (C-native, 4-8 bytes/element) instead of Python lists (28 bytes/element). Each batch is converted to CSR immediately and raw arrays are freed.

### solve_chunks

LSQR solver that operates on a `LinearOperator` built from column chunks. The matvec/rmatvec operations iterate over chunks sequentially, never needing the full matrix in memory.

## Requirements

- Python >= 3.8
- NumPy >= 1.20
- SciPy >= 1.7

## License

MIT License - Carmen Esteban, 2025-2026