tokenscope 0.2.0
TokenScope: Token-Aware Directory Explorer for LLMs
License: MIT
Requires Python: >=3.10
Keywords: mcp, directory, explorer, claude, ai, assistant, token-aware

# TokenScope

Token-Aware Directory Explorer for Large Language Models (LLMs).

A [Model Context Protocol (MCP)](https://modelcontextprotocol.io/) server that helps LLMs efficiently explore and understand codebases and directory structures.

## Overview

TokenScope provides intelligent directory structure analysis and token-aware file content exploration designed for LLMs like Claude. It helps LLMs understand codebases by:

1. Exploring directory structures with token-aware summarization
2. Viewing file contents with token limitations in mind
3. Generating comprehensive reports about directories

## Key Features

### Token-Aware Directory Exploration

- **Automatic summarization** for large directories while showing small directories in full
- **Respect for token limits** to maximize useful information within constraints
- **Built-in security** with base path validation
- **Smart filtering** with default patterns and .gitignore support
- **Accurate directory statistics** for even the largest directories

### Simple, Intuitive Tools

TokenScope provides just three core tools:

1. `explore_directory` - Scan and understand directory structures
2. `view_content` - Access file contents with token awareness
3. `generate_report` - Create comprehensive reports (with option to save to file)

## Installation

### Prerequisites

- Python 3.10 or higher
- [uv](https://github.com/astral-sh/uv) (recommended for dependency management)

### Installation (PyPI)

This is the recommended method for most users who just want to use TokenScope:

```bash
# Install from PyPI using uv (recommended)
uv pip install tokenscope
```

## Running TokenScope

The `--base-path` argument is mandatory for security reasons. It restricts all file operations to the specified directory.

```bash
# Run using the installed package
uv run --with tokenscope tokenscope --base-path /path/to/allowed/directory
```

### Configuring in Claude Desktop

1. Locate Claude Desktop's configuration file (typically `~/.config/claude/config.json`)

2. Add TokenScope to the `mcpServers` section:

  ```json
  "mcpServers": {
    "TokenScope": {
      "command": "uv",
      "args": [
        "run",
        "--with",
        "tokenscope",
        "tokenscope",
        "--base-path",
        "/your/secure/base/path"
      ]
    }
  }
  ```

3. Replace `/your/secure/base/path` with the directory you want to restrict operations to

4. Save the configuration file and restart Claude Desktop


## Usage

### Running TokenScope Server

The `--base-path` argument is required for security (it restricts file operations to the specified directory):

```bash
tokenscope --base-path /path/to/allowed/directory
```

### Testing Tools Directly

TokenScope includes a test mode for trying out tools directly:

```bash
# Test directory exploration
tokenscope --base-path /path/to/allowed/directory --test "explore:/path/to/directory"

# Test with custom ignore patterns
tokenscope --base-path /path/to/allowed/directory --test "explore:/path/to/directory?ignore=cache,*.log,tmp/&gitignore=false"

# Test file viewing
tokenscope --base-path /path/to/allowed/directory --test "view:/path/to/file"

# Test report generation
tokenscope --base-path /path/to/allowed/directory --test "report:/path/to/directory"

# Test report generation with file output and custom ignore patterns
tokenscope --base-path /path/to/allowed/directory --test "report:/path/to/directory?ignore=*.bak,temp/ > /path/to/output.md"
```

## Example Prompts

Here are some examples of how to use TokenScope with Claude:

```text
Could you explore my project directory at /path/to/project and tell me about its structure?
```

```text
Can you show me the content of the file at /path/to/file.py?
```

```text
Please generate a comprehensive report about my project at /path/to/project and save it to /path/to/report.md
```

## Accurate Directory Statistics

TokenScope now provides two levels of directory information:

1. **Quick Scan Statistics**: Information about files and directories visible in the tree view
2. **Full Directory Statistics**: Complete counts of ALL files and directories, even in very large repositories

This dual approach ensures that even for massive directories (with thousands or millions of files), you'll get accurate information about the total number of files, directories, and disk usage. This is especially valuable when working with large codebases where a complete directory listing would exceed token limits.

### Example Output

```txt
QUICK SCAN STATISTICS (files visible in tree):
Files shown in tree: 47
Directories shown in tree: 16
Size shown in tree: 185.9 MB

FULL DIRECTORY STATISTICS (all files):
Total files: 16,059
Total directories: 8
Total disk size: 2.1 GB
```

## Smart Filtering with Ignore Patterns

TokenScope automatically filters out common directories and files that typically don't contribute to understanding a codebase:

- **Default ignored patterns**: `.git/`, `.venv/`, `venv/`, `__pycache__/`, `node_modules/`, `build/`, `dist/`, etc.
- **Custom ignore patterns**: You can specify additional patterns to ignore via the `ignore_patterns` parameter
- **.gitignore support**: TokenScope can automatically respect .gitignore files in the directories it scans

This functionality helps prevent token waste on irrelevant files and directories like:

- Dependency directories (e.g., `node_modules`, virtual environments)
- Build artifacts and cache directories
- Version control metadata
- IDE configuration files

### Using Ignore Patterns in CLI Test Mode

```bash
# Ignore specific patterns
tokenscope --base-path /path --test "explore:/code?ignore=*.log,temp/"

# Disable .gitignore processing
tokenscope --base-path /path --test "explore:/code?gitignore=false"

# Both together
tokenscope --base-path /path --test "explore:/code?ignore=*.tmp&gitignore=false"
```

## Security Features

TokenScope includes important security features:

- All file operations are validated to ensure they're within the specified base directory
- Attempts to access files outside the base path are rejected
- The base path is set once when starting the server and cannot be changed without restart

## License

This project is licensed under the MIT License - see the LICENSE file for details.

## Acknowledgments

- Built with [FastMCP](https://github.com/jlowin/fastmcp)
- Uses [tiktoken](https://github.com/openai/tiktoken) for accurate token counting