index1 2.0.3
AI memory system for coding agents — code index + cognitive facts, persistent across sessions
Requires Python: >=3.10
Keywords: ai, memory, mcp, semantic-search, bm25, rag, cognitive, coding-agent

# index1

[English](README.md) | [中文](README.zh-CN.md)

AI memory system for coding agents. Code index + cognitive facts, persistent across sessions.

![index1 vs grep real-world comparison](assets/index1-ab-real.gif)

**index1 tested in real-world Claude grep!** Comparison of index1 + Claude grep vs Claude grep only:

https://github.com/user-attachments/assets/b689b0bb-b767-4fc8-9055-cc3ae872559e

## What is index1?

index1 gives AI coding agents **persistent memory** — two things LLMs lack natively:

| Module | Memory Type | What it stores |
|--------|-------------|---------------|
| **corpus/** | Semantic memory | Code index — files, functions, classes, structure |
| **cognition/** | Episodic memory | Cognitive facts — lessons learned, bug root causes, decisions made |

```
index1/
├── corpus/     Code index — what does the project look like?
└── cognition/  Cognitive facts — what did I learn last session?
```

## Install

**One-click** (recommended):

```bash
# macOS / Linux
curl -sSL https://raw.githubusercontent.com/gladego/index1/main/scripts/install.sh | bash

# Windows (PowerShell)
irm https://raw.githubusercontent.com/gladego/index1/main/scripts/install.ps1 | iex
```

The script auto-detects Python, installs via pipx, sets up Ollama, and creates default config.

**Manual install**:

```bash
pipx install index1    # recommended
# or: pip install index1
```

> **Note**: macOS blocks global `pip install` by default. Use [pipx](https://pipx.pypa.io/) instead:
> - macOS: `brew install pipx`
> - Linux: `pip install --user pipx && pipx ensurepath`
> - Windows: `scoop install pipx` or `pip install --user pipx`

## Quick Start

```bash
index1 setup                          # one-click Claude Code plugin install (hooks + MCP)
index1 index ./docs ./src             # index your project
index1 search "how does auth work"    # hybrid search
index1 web                            # open Web UI at localhost:6888
```

**No Ollama required** — index1 ships with built-in ONNX embedding (bge-small-en-v1.5, 384d). Ollama is optional for enhanced multilingual support.

## AI Tool Integration

### Claude Code

**One-click setup** (recommended):

```bash
index1 setup
```

This automatically registers hooks (SessionStart, PostToolUse, SessionEnd) and MCP server. Restart Claude Code — six tools become available:

| Tool | Description |
|------|-------------|
| `recall` | Unified search across code + facts |
| `learn` | Record an insight or decision |
| `read` | Read file content with index metadata |
| `status` | Index and cognition statistics |
| `reindex` | Rebuild project index |
| `config` | View/modify configuration |

**Manual setup** — add `.mcp.json` to your project root:

```json
{
  "mcpServers": {
    "index1": {
      "type": "stdio",
      "command": "index1",
      "args": ["serve"]
    }
  }
}
```

> Full setup guide: [Claude Code integration](docs/integration-claude-code.md) — MCP config, hooks, search strategy, CLAUDE.md setup

### Other AI Tools (OpenClaw, Cursor, Windsurf, Cline...)

**MCP-compatible tools**: Add the same `.mcp.json` config above to your tool's MCP settings.

**CLI mode** (works with any tool):

```bash
index1 search "how does authentication work"
index1 cog search "previous bug fixes"
```

> Full setup guide: [Other AI agents integration](docs/integration-other-agents.md)

## CLI Commands

```bash
# Core
index1 index <paths...>              # Index files/directories
index1 search <query>                # Hybrid search (BM25 + vector)
index1 status                        # View index & cognition statistics
index1 serve                         # Start MCP Server (stdio)
index1 web                           # Start Web UI (port 6888)

# Cognition
index1 cog search <query>            # Search cognitive facts
index1 cog list                      # List recent facts
index1 cog stats                     # Cognition statistics

# Setup & Maintenance
index1 setup                         # Install Claude Code plugin (hooks + MCP)
index1 doctor                        # Diagnose environment & health
index1 init                          # Configure embedding backend
index1 config [key] [value]          # View/modify configuration

# Advanced
index1 watch <paths...>              # Watch files for auto-reindex
index1 backfill                      # Backfill missing vectors
index1 repo-map                      # Generate project structure map
index1 symbols                       # View symbol statistics
```

## Embedding Backends

index1 supports multiple embedding backends with automatic fallback:

```
ONNX (default, built-in) → Ollama (optional) → BM25-only (always available)
```

| Backend | Model | Dim | Setup | Best for |
|---------|-------|-----|-------|----------|
| **ONNX** (default) | bge-small-en-v1.5 | 384 | Zero config | English, works offline, no external deps |
| Ollama | nomic-embed-text | 768 | `ollama pull nomic-embed-text` | Multilingual, higher accuracy |
| Ollama | bge-m3 | 1024 | `ollama pull bge-m3` | Chinese-optimized, 100+ languages |

Switch backend:

```bash
index1 init              # interactive backend selection
# or manually:
index1 config embed_backend ollama
index1 config embedding_model nomic-embed-text
```

> Without any embedding backend, index1 falls back to BM25-only keyword search. ONNX is bundled by default so vector search works out of the box.

## Supported File Types

**Structure-aware chunking** (language-specific parsers):

`.md` `.markdown` — headings-based splitting
`.py` — AST-based (functions, classes, methods)
`.rs` — regex pattern matching (fn, impl, struct, enum)
`.js` `.ts` `.jsx` `.tsx` — regex pattern matching (function, class, const)

**Generic chunking** (text-based splitting):

`.txt` `.text` `.yaml` `.yml` `.toml` `.json` `.cfg` `.ini` `.conf` `.sh` `.bash` `.html` `.css`

## Architecture

```
                    ┌─────────────────────────────────────┐
                    │          index1 MCP Server           │
Claude Code ──────► │                                     │
  (hooks +          │  ┌─────────┐    ┌──────────────┐    │
   MCP tools)       │  │ corpus  │    │  cognition   │    │
                    │  │         │    │              │    │
CLI ────────────►   │  │index.db │    │cognitive.db  │    │
                    │  │ FTS5    │    │ FTS5         │    │
Web UI ─────────►   │  │ vec     │    │ vec          │    │
                    │  └────┬────┘    └──────┬───────┘    │
                    │       │               │             │
                    │       └───────┬───────┘             │
                    │               │                     │
                    │     ONNX / Ollama Embedding         │
                    └─────────────────────────────────────┘

Hooks:
  SessionStart  → awaken (restore context)
  PostToolUse   → observe (capture facts from tool output)
  SessionEnd    → reflect (maintenance + consolidation)
```

- **Storage**: Dual SQLite databases — `index.db` (corpus) + `cognitive.db` (cognition)
- **Search**: BM25 + vector with Reciprocal Rank Fusion (RRF, k=60)
- **Chunking**: Structure-aware splitting by file type (AST/regex/headings)
- **Embedding**: ONNX (default, built-in) or Ollama (optional, multilingual)
- **Resilience**: safe_db_write, CircuitBreaker, ServiceCooldown, mcp_error_boundary

## Configuration

Config file: `~/.claude-index1/config.yaml`

```yaml
embed_backend: onnx                   # "onnx" (default) | "ollama"
onnx_model: BAAI/bge-small-en-v1.5    # ONNX model
embedding_model: nomic-embed-text     # Ollama model (when embed_backend=ollama)
embedding_dim: 768
ollama_url: http://localhost:11434
top_k: 10                             # Results per query
collection: default                   # Namespace isolation
web_port: 6888                        # Web UI port
```

Project-level override: `.index1.yaml` in project root.

Environment variable: `INDEX1_HOME` overrides default `~/.claude-index1/` directory.

## Performance

| Mode | Cold | Hot (cached) |
|------|------|-------------|
| Hybrid (BM25 + Vector) | 40–180 ms | < 1 ms |
| BM25-only (no embedding) | ~35 ms | < 1 ms |
| Grep/Glob (native) | 4 ms | N/A |

**Context savings**: index1 returns top-k ranked results (~400–500 tokens) vs Grep returning all matches (~5,000–35,000 tokens for common keywords). Saves **90–99% of LLM context window** on broad queries.

Full benchmark: [Benchmark: index1 vs native tools](docs/benchmark-vs-native-tools.md) ([中文](docs/benchmark-vs-native-tools.zh-CN.md))

## FAQ

**Do I need Ollama?**
No. index1 v2 ships with built-in ONNX embedding — vector search works out of the box. Ollama is optional for enhanced multilingual and Chinese support.

**Can I use multiple projects?**
Yes. Use `--collection` to isolate namespaces:

```bash
index1 index ./project-a -c proj_a
index1 search "query" -c proj_a
```

With Claude Code hooks, collection is auto-detected from project directory.

**Where is data stored?**
- Corpus: `~/.claude-index1/index.db`
- Cognition: `~/.claude-index1/cognitive.db`
- Config: `~/.claude-index1/config.yaml`

Override with `INDEX1_HOME` environment variable.

**How to rebuild the index?**

```bash
index1 index --force ./docs ./src
```

**How to check system health?**

```bash
index1 doctor
```

## Contributing

```bash
git clone https://github.com/gladego/index1.git
cd index1
pip install -e ".[dev]"
pytest
```

PRs welcome. Please ensure `pytest` passes before submitting.

## Changelog

### v2.0.0

- **Cognition module** — episodic memory with cognitive facts (learn, recall, maintenance)
- **Dual database** — separate index.db (corpus) + cognitive.db (cognition)
- **ONNX default embedding** — built-in bge-small-en-v1.5, zero external deps for vector search
- **Claude Code plugin system** — `index1 setup` one-click install (hooks + MCP)
- **6 MCP tools** — recall, learn, read, status, reindex, config (unified search across code + facts)
- **Hook-based fact capture** — SessionStart/PostToolUse/SessionEnd automatic observation
- **Memory layering** — Ebbinghaus decay, layer promotion (L3b → L3a → L2), pattern extraction
- **Maintenance pipeline** — 10-step background consolidation (embedding, dedup, decay, invalidation)
- **Observer Worker** — persistent LLM observer for deep fact extraction
- **Web UI** — dashboard with corpus, cognition, and memory tabs
- **Resilience** — safe_db_write, CircuitBreaker, ServiceCooldown, mcp_error_boundary
- **Query understanding** — intent detection, symbol lookup, query expansion
- **tree-sitter code graph** — repo-map with symbol extraction

### v0.1.0

- BM25 + vector hybrid search with RRF fusion
- Structure-aware chunking (Markdown, Python, Rust, JS/TS)
- MCP Server with 5 tools for Claude Code integration
- Web UI with Atom Core animated logo
- L1/L2 query cache (10min TTL)
- File watcher for auto-reindex
- One-click install script

## Requirements

- Python >= 3.10
- macOS / Linux / Windows
- ONNX embedding built-in (bge-small-en-v1.5) — vector search works out of the box
- [Ollama](https://ollama.ai) (optional, for enhanced multilingual/CJK support)

## License

[Apache 2.0](LICENSE)