mizan-rag 0.1.1
A complete RAG pipeline built on Mizan Similarity, MizanEmbedder, and MizanVector.
License: MIT
Requires Python: >=3.9

# Mizan RAG

A production‑ready, scale‑aware Retrieval‑Augmented Generation (RAG) framework powered by the **Mizan Balance Function**, a new geometric similarity metric designed for robust vector search, balanced embeddings, and improved retrieval accuracy.

Mizan RAG integrates:
- **Mizan Similarity** (your custom metric)
- **Mizan Embedding Models**
- **Mizan Vector Search**
- **Multi‑provider LLMs (OpenAI, Groq, Grok, HF, Local)**
- **Chunking, retrieval, re-ranking, summarization**
- **Caching, performance optimizations, and extensible architecture**

This README is the **full expanded version** (A + B + C + D combined), built for:
- Developers  
- Researchers  
- Students  
- Industry teams  
- Open‑source contributors  

---

# 🚀 Features

### Scale-Aware Retrieval (Cosine + Mizan)
- Replace cosine similarity with Mizan for robust retrieval.
- Fixes chunk imbalance, scale distortion, and embedding collapse.

### Full RAG Pipeline
- Chunker  
- Retriever  
- Mizan Ranker (cosine/mizan/hybrid)  
- Summarizer (multi-provider LLM)  
- Embedding wrapper  
- End-to-end orchestration  

### Cache System
- Embedding cache  
- Index cache  
- LLM response cache  

### Supports Multiple LLM Providers
- **Groq (recommended)**  
- OpenAI  
- Grok (xAI)  
- OpenRouter  
- HuggingFace  
- Local HF models  

### Extensible, production-ready folder structure

---

# Installation

### 1. Clone repository
```bash
git clone https://github.com/ahsanshaokat/mizan-rag.git
cd mizan-rag
```

### 2. Install dependencies
```bash
pip install -r requirements.txt
```

### 3. Install Mizan ecosystem packages (if separate repos)
```bash
pip install mizan_vector mizansimilarity mizan-embedder
```

### 4. Install backend models (optional)
```bash
pip install sentence-transformers
pip install transformers
```

---

#  Quick Start

### 1. Put your `.txt` files in:
```
/docs/
```

### 2. Run:
```bash
python main.py
```

### 3. Ask questions:
```
Ask your question (or 'exit'): What is the Mizan Balance Function?
```

---

#  Mizan Balance Function (Short Overview)

Mizan is a **scale-aware similarity metric** designed to fix weaknesses in cosine similarity:

| Issue in Cosine | Mizan Fix |
|-----------------|-----------|
| Loses magnitude information | Preserves proportion & balance |
| Weak separation in high dimensions | Stronger scaling geometry |
| Chunk length bias | Scale normalization |
| Embedding collapse | Balanced magnitude alignment |

Mizan Similarity Formula (simplified):
```
miz(a, b) = (directional similarity * balance score)
```

Full math is in the Mizan research paper.

---

#  Architecture Overview

```
docs/                  → Your text files
mizan_rag/
  ├── chunker.py       → Word-based intelligent chunking
  ├── retriever.py     → Dense retrieval (cosine/mizan)
  ├── ranker.py        → Second-pass reranking (cosine/mizan/hybrid)
  ├── summarizer.py    → LLM integration (Groq, OpenAI, etc.)
  ├── utils/cache.py   → Persistent cache engine
  ├── pipeline.py      → Full RAG orchestration
main.py                → CLI interface
```

---

#  Pipeline Stages

### **1. Chunking**
- Word-based  
- Configurable size + overlap  
- Clean and lightweight  

### **2. Embedding**
Uses your trained or HF model:
```
sentence-transformers/all-MiniLM-L6-v2  (recommended for Mizan)
```

### **3. Retriever**
- Computes embeddings  
- Uses cosine or Mizan similarity  
- Ultra-fast search  
- Cached for performance  

### **4. Ranker**
- Optional re-ranking using:
  - cosine  
  - mizan  
  - hybrid(w1 * cosine + w2 * mizan)

### **5. Summarizer**
Supports:
- Groq
- OpenAI
- Grok
- OpenRouter
- HuggingFace
- Local models

---

# 🛠 Example: Using MizanRetriever

```python
from mizan_rag.retriever import MizanRetriever

retriever = MizanRetriever(embed_fn, cache)
results = retriever.search("What is Mizan?", top_k=5, metric="mizan")
```

---

#  Example: Full RAG Query

```python
result = rag.query(
    "Explain the Mizan Balance Function",
    top_k_retrieve=8,
    top_k_rerank=5
)

print(result["answer"])
```

---

# ⚡ Recommended Embedding Models for Mizan

### ⭐ Best (Highly compatible)
- all-MiniLM-L6-v2  
- distilbert-base-uncased  
- paraphrase-MiniLM-L3-v2  

### ⚠️ Avoid (collapsed or incompatible geometry)
- all-mpnet-base-v2  
- intfloat/e5-base  
- e5-small-v2  
- instructor-base / large  

---

#  Documentation Roadmap (Full 18‑Article Series)

This README is part of a larger ecosystem:

## SECTION 1 — Foundations
1. Mizan Balance Function  
2. Why Cosine Fails  
3. Visualizing Metrics  
4. Mathematics Behind Mizan  

## SECTION 2 — Engineering
5. Using MizanSimilarity  
6. Building RAG With Mizan  
7. One Change That Improves Retrieval  

## SECTION 3 — Embedding Models
8. Designing Mizan Embedding Model  
9. Training MizanTextEncoder  
10. MizanContrastiveLoss  
11. Benchmarking Mizan  

## SECTION 4 — Mizan Ecosystem
12. MizanVector  
13. MizanEmbedder  
14. MizanModels  

## SECTION 5 — Industry
15. Enterprise RAG Failures  
16. Search, Chatbots, Recommendations  
17. Multi-modal Mizan  

## SECTION 6 — Research
18. Formal Mathematical Treatment of Mizan  

---

# 🧪 Simple Test Query

Try:

```
What does the Quran say about balance?
What is embedding magnitude?
Who is Ahsan Shaokat?
Explain vector projection.
How does Mizan fix cosine issues?
```

These match the synthetic dataset perfectly.

---

# 📄 License
MIT License.

---

# 🤝 Contributions
PRs welcome.  
Feature requests encouraged.  
We are building an open ecosystem around **scale-aware AI**.

---

# ⭐ Star the Repository
If this library helps you, please give it a star ✨