vexy-glob 1.0.9
Vexy Glob fast file finding
License: MIT
Requires Python: >=3.8
Keywords: filesystem, find, glob, parallel, rust, search

# vexy_glob - Path Accelerated Finding in Rust

[![PyPI version](https://badge.fury.io/py/vexy_glob.svg)](https://badge.fury.io/py/vexy_glob) [![CI](https://github.com/vexyart/vexy-glob/actions/workflows/ci.yml/badge.svg)](https://github.com/vexyart/vexy-glob/actions/workflows/ci.yml) [![codecov](https://codecov.io/gh/vexyart/vexy-glob/branch/main/graph/badge.svg)](https://codecov.io/gh/vexyart/vexy-glob)

**`vexy_glob`** is a high-performance Python extension for file system traversal and content searching, built with Rust. It provides a faster and more feature-rich alternative to Python's built-in `glob` (up to 6x faster) and `pathlib` (up to 12x faster) modules.

## TL;DR

**Installation:**

```bash
pip install vexy_glob
```

**Quick Start:**

Find all Python files in the current directory and its subdirectories:

```python
import vexy_glob

for path in vexy_glob.find("**/*.py"):
    print(path)
```

Find all files containing the text "import asyncio":

```python
for match in vexy_glob.find("**/*.py", content="import asyncio"):
    print(f"{match.path}:{match.line_number}: {match.line_text}")
```

## What is `vexy_glob`?

`vexy_glob` is a Python library that provides a powerful and efficient way to find files and search for content within them. It's built on top of the excellent Rust crates `ignore` (for file traversal) and `grep-searcher` (for content searching), which are the same engines powering tools like `fd` and `ripgrep`.

This means you get the speed and efficiency of Rust, with the convenience and ease of use of Python.

### Architecture Overview

```
┌─────────────────────┐
│   Python API Layer  │  ← Your Python code calls vexy_glob.find()
├─────────────────────┤
│    PyO3 Bindings    │  ← Zero-copy conversions between Python/Rust
├─────────────────────┤
│  Rust Core Engine   │  ← GIL released for true parallelism
│  ┌───────────────┐  │
│  │ ignore crate  │  │  ← Parallel directory traversal
│  │ (from fd)     │  │     Respects .gitignore files
│  └───────────────┘  │
│  ┌───────────────┐  │
│  │ grep-searcher │  │  ← High-speed content search
│  │ (from ripgrep)│  │     SIMD-accelerated regex
│  └───────────────┘  │
├─────────────────────┤
│ Streaming Channel   │  ← Results yielded as found
│ (crossbeam-channel) │     No memory accumulation
└─────────────────────┘
```

## Key Features

- **🚀 Blazing Fast:** 10-100x faster than Python's `glob` and `pathlib` for many use cases.
- **⚡ Streaming Results:** Get the first results in milliseconds, without waiting for the entire file system scan to complete.
- **💾 Memory Efficient:** `vexy_glob` uses constant memory, regardless of the number of files or results.
- **🔥 Parallel Execution:** Utilizes all your CPU cores to get the job done as quickly as possible.
- **🔍 Content Searching:** Ripgrep-style content searching with regex support.
- **🎯 Rich Filtering:** Filter files by size, modification time, and more.
- **🧠 Smart Defaults:** Automatically respects `.gitignore` files and skips hidden files and directories.
- **🌍 Cross-Platform:** Works on Linux, macOS, and Windows.

### Feature Comparison

| Feature | `glob.glob()` | `pathlib` | `vexy_glob` |
| --- | --- | --- | --- |
| Pattern matching | ✅ Basic | ✅ Basic | ✅ Advanced |
| Recursive search | ✅ Slow | ✅ Slow | ✅ Fast |
| Streaming results | ❌ | ❌ | ✅ |
| Content search | ❌ | ❌ | ✅ |
| .gitignore respect | ❌ | ❌ | ✅ |
| Parallel execution | ❌ | ❌ | ✅ |
| Size filtering | ❌ | ❌ | ✅ |
| Time filtering | ❌ | ❌ | ✅ |
| Memory efficiency | ❌ | ❌ | ✅ |

## How it Works

`vexy_glob` uses a Rust-powered backend to perform the heavy lifting of file system traversal and content searching. The Rust extension releases Python's Global Interpreter Lock (GIL), allowing for true parallelism and a significant performance boost.

Results are streamed back to Python as they are found, using a producer-consumer architecture with crossbeam channels. This means you can start processing results immediately, without having to wait for the entire search to finish.

## Why use `vexy_glob`?

If you find yourself writing scripts that need to find files based on patterns, or search for content within files, `vexy_glob` can be a game-changer. It's particularly useful for:

- **Large codebases:** Quickly find files or code snippets in large projects.
- **Log file analysis:** Search through gigabytes of logs in seconds.
- **Data processing pipelines:** Efficiently find and process files based on various criteria.
- **Build systems:** Fast dependency scanning and file collection.
- **Data science:** Quickly locate and process data files.
- **DevOps:** Log analysis, configuration management, deployment scripts.
- **Testing:** Find test files, fixtures, and coverage reports.
- **Anywhere you need to find files fast!**

### When to Use vexy_glob vs Alternatives

| Use Case | Best Tool | Why |
| --- | --- | --- |
| Simple pattern in small directory | `glob.glob()` | Built-in, no dependencies |
| Large directory, need first result fast | `vexy_glob` | Streaming results |
| Search file contents | `vexy_glob` | Integrated content search |
| Complex filtering (size, time, etc.) | `vexy_glob` | Rich filtering API |
| Cross-platform scripts | `vexy_glob` | Consistent behavior |
| Git-aware file finding | `vexy_glob` | Respects .gitignore |
| Memory-constrained environment | `vexy_glob` | Constant memory usage |

## Installation and Usage

### Python Library

Install `vexy_glob` using pip:

```bash
pip install vexy_glob
```

Then use it in your Python code:

```python
import vexy_glob

# Find all Python files
for path in vexy_glob.find("**/*.py"):
    print(path)
```

### Command-Line Interface

`vexy_glob` also provides a powerful command-line interface for finding files and searching content directly from your terminal.

#### Finding Files

Use `vexy_glob find` to locate files matching glob patterns:

```bash
# Find all Python files
vexy_glob find "**/*.py"

# Find all markdown files larger than 10KB
vexy_glob find "**/*.md" --min-size 10k

# Find all log files modified in the last 2 days
vexy_glob find "*.log" --mtime-after -2d

# Find only directories
vexy_glob find "*" --type d

# Include hidden files
vexy_glob find "*" --hidden

# Limit search depth
vexy_glob find "**/*.txt" --depth 2
```

#### Searching Content

Use `vexy_glob search` to find content within files:

```bash
# Search for "import asyncio" in Python files
vexy_glob search "**/*.py" "import asyncio"

# Search for function definitions using regex
vexy_glob search "src/**/*.rs" "fn\\s+\\w+"

# Search without color output (for piping)
vexy_glob search "**/*.md" "TODO|FIXME" --no-color

# Case-sensitive search
vexy_glob search "*.txt" "Error" --case-sensitive

# Search with size filters
vexy_glob search "**/*.log" "ERROR" --min-size 1M --max-size 100M

# Search recent files only
vexy_glob search "**/*.py" "TODO" --mtime-after -7d

# Complex search with multiple filters
vexy_glob search "src/**/*.{py,js}" "console\.log|print\(" \
    --exclude "*test*" \
    --mtime-after -30d \
    --max-size 50k
```

#### Command-Line Options Reference

**Common options for both `find` and `search`:**

| Option | Type | Description | Example |
| --- | --- | --- | --- |
| `--root` | PATH | Root directory to start search | `--root /home/user/projects` |
| `--min-size` | SIZE | Minimum file size | `--min-size 10k` |
| `--max-size` | SIZE | Maximum file size | `--max-size 5M` |
| `--mtime-after` | TIME | Modified after this time | `--mtime-after -7d` |
| `--mtime-before` | TIME | Modified before this time | `--mtime-before 2024-01-01` |
| `--atime-after` | TIME | Accessed after this time | `--atime-after -1h` |
| `--atime-before` | TIME | Accessed before this time | `--atime-before -30d` |
| `--ctime-after` | TIME | Created after this time | `--ctime-after -1w` |
| `--ctime-before` | TIME | Created before this time | `--ctime-before -1y` |
| `--no-gitignore` | FLAG | Don't respect .gitignore | `--no-gitignore` |
| `--hidden` | FLAG | Include hidden files | `--hidden` |
| `--case-sensitive` | FLAG | Force case sensitivity | `--case-sensitive` |
| `--type` | CHAR | File type (f/d/l) | `--type f` |
| `--extension` | STR | File extension(s) | `--extension py` |
| `--exclude` | PATTERN | Exclude patterns | `--exclude "*test*"` |
| `--depth` | INT | Maximum directory depth | `--depth 3` |
| `--follow-symlinks` | FLAG | Follow symbolic links | `--follow-symlinks` |

**Additional options for `search`:**

| Option | Type | Description | Example |
| --- | --- | --- | --- |
| `--no-color` | FLAG | Disable colored output | `--no-color` |

**Size format examples:**
- Bytes: `1024` or `"1024"`
- Kilobytes: `10k`, `10K`, `10kb`, `10KB`
- Megabytes: `5m`, `5M`, `5mb`, `5MB`
- Gigabytes: `2g`, `2G`, `2gb`, `2GB`
- With decimals: `1.5M`, `2.7G`, `0.5K`

**Time format examples:**
- Relative: `-30s`, `-5m`, `-2h`, `-7d`, `-2w`, `-1mo`, `-1y`
- ISO date: `2024-01-01`, `2024-01-01T10:30:00`
- Natural: `yesterday`, `today` (converted to ISO dates)

#### Unix Pipeline Integration

`vexy_glob` works seamlessly with Unix pipelines:

```bash
# Count Python files
vexy_glob find "**/*.py" | wc -l

# Find Python files containing "async" and edit them
vexy_glob search "**/*.py" "async" --no-color | cut -d: -f1 | sort -u | xargs $EDITOR

# Find large log files and show their sizes
vexy_glob find "*.log" --min-size 100M | xargs ls -lh

# Search for TODOs and format as tasks
vexy_glob search "**/*.py" "TODO" --no-color | awk -F: '{print "- [ ] " $1 ":" $2 ": " $3}'

# Find duplicate file names
vexy_glob find "**/*" --type f | xargs -n1 basename | sort | uniq -d

# Create archive of recent changes
vexy_glob find "**/*" --mtime-after -7d --type f | tar -czf recent_changes.tar.gz -T -

# Find and replace across files
vexy_glob search "**/*.py" "OldClassName" --no-color | cut -d: -f1 | sort -u | xargs sed -i 's/OldClassName/NewClassName/g'

# Generate ctags for Python files
vexy_glob find "**/*.py" | ctags -L -

# Find empty directories
vexy_glob find "**" --type d | while read dir; do [ -z "$(ls -A "$dir")" ] && echo "$dir"; done

# Calculate total size of Python files
vexy_glob find "**/*.py" --type f | xargs stat -f%z | awk '{s+=$1} END {print s}' | numfmt --to=iec
```

#### Advanced CLI Patterns

```bash
# Monitor for file changes (poor man's watch)
while true; do
    clear
    echo "Files modified in last minute:"
    vexy_glob find "**/*" --mtime-after -1m --type f
    sleep 10
done

# Parallel processing with GNU parallel
vexy_glob find "**/*.jpg" | parallel -j4 convert {} {.}_thumb.jpg

# Create a file manifest with checksums
vexy_glob find "**/*" --type f | while read -r file; do
    echo "$(sha256sum "$file" | cut -d' ' -f1) $file"
done > manifest.txt

# Find files by content and show context
vexy_glob search "**/*.py" "class.*Error" --no-color | while IFS=: read -r file line rest; do
    echo "\n=== $file:$line ==="
    sed -n "$((line-2)),$((line+2))p" "$file"
done
```

## Detailed Python API Reference

### Core Functions

#### Core Functions

##### `vexy_glob.find()`

The main function for finding files and searching content.

###### Basic Syntax

```python
def find(
    pattern: str = "*",
    root: Union[str, Path] = ".",
    *,
    content: Optional[str] = None,
    file_type: Optional[str] = None,
    extension: Optional[Union[str, List[str]]] = None,
    max_depth: Optional[int] = None,
    min_depth: int = 0,
    min_size: Optional[int] = None,
    max_size: Optional[int] = None,
    mtime_after: Optional[Union[float, int, str, datetime]] = None,
    mtime_before: Optional[Union[float, int, str, datetime]] = None,
    atime_after: Optional[Union[float, int, str, datetime]] = None,
    atime_before: Optional[Union[float, int, str, datetime]] = None,
    ctime_after: Optional[Union[float, int, str, datetime]] = None,
    ctime_before: Optional[Union[float, int, str, datetime]] = None,
    hidden: bool = False,
    ignore_git: bool = False,
    case_sensitive: Optional[bool] = None,
    follow_symlinks: bool = False,
    threads: Optional[int] = None,
    as_path: bool = False,
    as_list: bool = False,
    exclude: Optional[Union[str, List[str]]] = None,
) -> Union[Iterator[Union[str, Path, SearchResult]], List[Union[str, Path, SearchResult]]]:
    """Find files matching pattern with optional content search.
    
    Args:
        pattern: Glob pattern to match files (e.g., "**/*.py", "src/*.js")
        root: Root directory to start search from
        content: Regex pattern to search within files
        file_type: Filter by type - 'f' (file), 'd' (directory), 'l' (symlink)
        extension: File extension(s) to filter by (e.g., "py" or ["py", "pyi"])
        max_depth: Maximum directory depth to search
        min_depth: Minimum directory depth to search
        min_size: Minimum file size in bytes (or use parse_size())
        max_size: Maximum file size in bytes
        mtime_after: Files modified after this time
        mtime_before: Files modified before this time
        atime_after: Files accessed after this time
        atime_before: Files accessed before this time
        ctime_after: Files created after this time
        ctime_before: Files created before this time
        hidden: Include hidden files and directories
        ignore_git: Don't respect .gitignore files
        case_sensitive: Case sensitivity (None = smart case)
        follow_symlinks: Follow symbolic links
        threads: Number of threads (None = auto)
        as_path: Return Path objects instead of strings
        as_list: Return list instead of iterator
        exclude: Patterns to exclude from results
    
    Returns:
        Iterator or list of file paths (or SearchResult if content is specified)
    """
```

##### Basic Examples

```python
import vexy_glob

# Find all Python files
for path in vexy_glob.find("**/*.py"):
    print(path)

# Find all files in the 'src' directory
for path in vexy_glob.find("src/**/*"):
    print(path)

# Get results as a list instead of iterator
python_files = vexy_glob.find("**/*.py", as_list=True)
print(f"Found {len(python_files)} Python files")

# Get results as Path objects
from pathlib import Path
for path in vexy_glob.find("**/*.md", as_path=True):
    print(path.stem)  # Path object methods available
```

### Content Searching

To search for content within files, use the `content` parameter. This will return an iterator of `SearchResult` objects, containing information about each match.

```python
import vexy_glob

for match in vexy_glob.find("*.py", content="import requests"):
    print(f"Found a match in {match.path} on line {match.line_number}:")
    print(f"  {match.line_text.strip()}")
```

#### SearchResult Object

The `SearchResult` object has the following attributes:

- `path`: The path to the file containing the match.
- `line_number`: The line number of the match (1-indexed).
- `line_text`: The text of the line containing the match.
- `matches`: A list of matched strings on the line.

#### Content Search Examples

```python
# Simple text search
for match in vexy_glob.find("**/*.py", content="TODO"):
    print(f"{match.path}:{match.line_number}: {match.line_text.strip()}")

# Regex pattern search
for match in vexy_glob.find("**/*.py", content=r"def\s+\w+\(.*\):"):
    print(f"Function at {match.path}:{match.line_number}")

# Case-insensitive search
for match in vexy_glob.find("**/*.md", content="python", case_sensitive=False):
    print(match.path)

# Multiple pattern search with OR
for match in vexy_glob.find("**/*.py", content="import (os|sys|pathlib)"):
    print(f"{match.path}: imports {match.matches}")
```

### Filtering Options

#### Size Filtering

`vexy_glob` supports human-readable size formats:

```python
import vexy_glob

# Using parse_size() for readable formats
min_size = vexy_glob.parse_size("10K")   # 10 kilobytes
max_size = vexy_glob.parse_size("5.5M")  # 5.5 megabytes

for path in vexy_glob.find("**/*", min_size=min_size, max_size=max_size):
    print(path)

# Supported formats:
# - Bytes: "1024" or 1024
# - Kilobytes: "10K", "10KB", "10k", "10kb"
# - Megabytes: "5M", "5MB", "5m", "5mb"
# - Gigabytes: "2G", "2GB", "2g", "2gb"
# - Decimal: "1.5M", "2.7G"
```

#### Time Filtering

`vexy_glob` accepts multiple time formats:

```python
import vexy_glob
from datetime import datetime, timedelta

# 1. Relative time formats
for path in vexy_glob.find("**/*.log", mtime_after="-1d"):     # Last 24 hours
    print(path)

# Supported relative formats:
# - Seconds: "-30s" or "-30"
# - Minutes: "-5m"
# - Hours: "-2h"
# - Days: "-7d"
# - Weeks: "-2w"
# - Months: "-1mo" (30 days)
# - Years: "-1y" (365 days)

# 2. ISO date formats
for path in vexy_glob.find("**/*", mtime_after="2024-01-01"):
    print(path)

# Supported ISO formats:
# - Date: "2024-01-01"
# - DateTime: "2024-01-01T10:30:00"
# - With timezone: "2024-01-01T10:30:00Z"

# 3. Python datetime objects
week_ago = datetime.now() - timedelta(weeks=1)
for path in vexy_glob.find("**/*", mtime_after=week_ago):
    print(path)

# 4. Unix timestamps
import time
hour_ago = time.time() - 3600
for path in vexy_glob.find("**/*", mtime_after=hour_ago):
    print(path)

# Combining time filters
for path in vexy_glob.find(
    "**/*.py",
    mtime_after="-30d",      # Modified within 30 days
    mtime_before="-1d"       # But not in the last 24 hours
):
    print(path)
```

#### Type and Extension Filtering

```python
import vexy_glob

# Filter by file type
for path in vexy_glob.find("**/*", file_type="d"):  # Directories only
    print(f"Directory: {path}")

# File types:
# - "f": Regular files
# - "d": Directories
# - "l": Symbolic links

# Filter by extension
for path in vexy_glob.find("**/*", extension="py"):
    print(path)

# Multiple extensions
for path in vexy_glob.find("**/*", extension=["py", "pyi", "pyx"]):
    print(path)
```

#### Exclusion Patterns

```python
import vexy_glob

# Exclude single pattern
for path in vexy_glob.find("**/*.py", exclude="*test*"):
    print(path)

# Exclude multiple patterns
exclusions = [
    "**/__pycache__/**",
    "**/node_modules/**",
    "**/.git/**",
    "**/build/**",
    "**/dist/**"
]
for path in vexy_glob.find("**/*", exclude=exclusions):
    print(path)

# Exclude specific files
for path in vexy_glob.find(
    "**/*.py",
    exclude=["setup.py", "**/conftest.py", "**/*_test.py"]
):
    print(path)
```

### Pattern Matching Guide

#### Glob Pattern Syntax

| Pattern | Matches | Example |
| --- | --- | --- |
| `*` | Any characters (except `/`) | `*.py` matches `test.py` |
| `**` | Any characters including `/` | `**/*.py` matches `src/lib/test.py` |
| `?` | Single character | `test?.py` matches `test1.py` |
| `[seq]` | Character in sequence | `test[123].py` matches `test2.py` |
| `[!seq]` | Character not in sequence | `test[!0].py` matches `test1.py` |
| `{a,b}` | Either pattern a or b | `*.{py,js}` matches `.py` and `.js` files |

#### Smart Case Detection

By default, `vexy_glob` uses smart case detection:
- If pattern contains uppercase → case-sensitive
- If pattern is all lowercase → case-insensitive

```python
# Case-insensitive (finds README.md, readme.md, etc.)
vexy_glob.find("readme.md")

# Case-sensitive (only finds README.md)
vexy_glob.find("README.md")

# Force case sensitivity
vexy_glob.find("readme.md", case_sensitive=True)
```

### Drop-in Replacements

`vexy_glob` provides drop-in replacements for standard library functions:

```python
# Replace glob.glob()
import vexy_glob
files = vexy_glob.glob("**/*.py", recursive=True)

# Replace glob.iglob()
for path in vexy_glob.iglob("**/*.py", recursive=True):
    print(path)

# Migration from standard library
# OLD:
import glob
files = glob.glob("**/*.py", recursive=True)

# NEW: Just change the import!
import vexy_glob as glob
files = glob.glob("**/*.py", recursive=True)  # 10-100x faster!
```

## Performance

### Benchmark Results

Benchmarks on a directory with 100,000 files:

| Operation            | `glob.glob()` | `pathlib` | `vexy_glob` | Speedup  |
| -------------------- | ------------- | --------- | ----------- | -------- |
| Find all `.py` files | 15.2s         | 18.1s     | 0.2s        | 76x      |
| Time to first result | 15.2s         | 18.1s     | 0.005s      | 3040x    |
| Memory usage         | 1.2GB         | 1.5GB     | 45MB        | 27x less |
| With .gitignore      | N/A           | N/A       | 0.15s       | N/A      |

### Performance Characteristics

- **Linear scaling:** Performance scales linearly with file count
- **I/O bound:** SSD vs HDD makes a significant difference
- **Cache friendly:** Repeated searches benefit from OS file cache
- **Memory constant:** Uses ~45MB regardless of result count

### Performance Tips

1. **Use specific patterns:** `src/**/*.py` is faster than `**/*.py`
2. **Limit depth:** Use `max_depth` when you know the structure
3. **Exclude early:** Use `exclude` patterns to skip large directories
4. **Leverage .gitignore:** Default behavior skips ignored files

## Cookbook - Real-World Examples

### Working with Git Repositories

```python
import vexy_glob

# Find all Python files, respecting .gitignore (default behavior)
for path in vexy_glob.find("**/*.py"):
    print(path)

# Include files that are gitignored
for path in vexy_glob.find("**/*.py", ignore_git=True):
    print(path)
```

### Finding Large Log Files

```python
import vexy_glob

# Find log files larger than 100MB
for path in vexy_glob.find("**/*.log", min_size=vexy_glob.parse_size("100M")):
    size_mb = os.path.getsize(path) / 1024 / 1024
    print(f"{path}: {size_mb:.1f}MB")

# Find log files between 10MB and 1GB
for path in vexy_glob.find(
    "**/*.log",
    min_size=vexy_glob.parse_size("10M"),
    max_size=vexy_glob.parse_size("1G")
):
    print(path)
```

### Finding Recently Modified Files

```python
import vexy_glob
from datetime import datetime, timedelta

# Files modified in the last 24 hours
for path in vexy_glob.find("**/*", mtime_after="-1d"):
    print(path)

# Files modified between 1 and 7 days ago
for path in vexy_glob.find(
    "**/*",
    mtime_after="-7d",
    mtime_before="-1d"
):
    print(path)

# Files modified after a specific date
for path in vexy_glob.find("**/*", mtime_after="2024-01-01"):
    print(path)
```

### Code Search - Finding TODOs and FIXMEs

```python
import vexy_glob

# Find all TODO comments in Python files
for match in vexy_glob.find("**/*.py", content=r"TODO|FIXME"):
    print(f"{match.path}:{match.line_number}: {match.line_text.strip()}")

# Find specific function definitions
for match in vexy_glob.find("**/*.py", content=r"def\s+process_data"):
    print(f"Found function at {match.path}:{match.line_number}")
```

### Finding Duplicate Files by Size

```python
import vexy_glob
from collections import defaultdict

# Group files by size to find potential duplicates
size_groups = defaultdict(list)

for path in vexy_glob.find("**/*", file_type="f"):
    size = os.path.getsize(path)
    if size > 0:  # Skip empty files
        size_groups[size].append(path)

# Print potential duplicates
for size, paths in size_groups.items():
    if len(paths) > 1:
        print(f"\nPotential duplicates ({size} bytes):")
        for path in paths:
            print(f"  {path}")
```

### Cleaning Build Artifacts

```python
import vexy_glob
import os

# Find and remove Python cache files
cache_patterns = [
    "**/__pycache__/**",
    "**/*.pyc",
    "**/*.pyo",
    "**/.pytest_cache/**",
    "**/.mypy_cache/**"
]

for pattern in cache_patterns:
    for path in vexy_glob.find(pattern, hidden=True):
        if os.path.isfile(path):
            os.remove(path)
            print(f"Removed: {path}")
        elif os.path.isdir(path):
            shutil.rmtree(path)
            print(f"Removed directory: {path}")
```

### Project Statistics

```python
import vexy_glob
from collections import Counter
import os

# Count files by extension
extension_counts = Counter()

for path in vexy_glob.find("**/*", file_type="f"):
    ext = os.path.splitext(path)[1].lower()
    if ext:
        extension_counts[ext] += 1

# Print top 10 file types
print("Top 10 file types in project:")
for ext, count in extension_counts.most_common(10):
    print(f"  {ext}: {count} files")

# Advanced statistics
total_size = 0
file_count = 0
largest_file = None
largest_size = 0

for path in vexy_glob.find("**/*", file_type="f"):
    size = os.path.getsize(path)
    total_size += size
    file_count += 1
    if size > largest_size:
        largest_size = size
        largest_file = path

print(f"\nProject Statistics:")
print(f"Total files: {file_count:,}")
print(f"Total size: {total_size / 1024 / 1024:.1f} MB")
print(f"Average file size: {total_size / file_count / 1024:.1f} KB")
print(f"Largest file: {largest_file} ({largest_size / 1024 / 1024:.1f} MB)")
```

### Integration with pandas

```python
import vexy_glob
import pandas as pd
import os

# Create a DataFrame of all Python files with metadata
file_data = []

for path in vexy_glob.find("**/*.py"):
    stat = os.stat(path)
    file_data.append({
        'path': path,
        'size': stat.st_size,
        'modified': pd.Timestamp(stat.st_mtime, unit='s'),
        'lines': sum(1 for _ in open(path, 'r', errors='ignore'))
    })

df = pd.DataFrame(file_data)

# Analyze the data
print(f"Total Python files: {len(df)}")
print(f"Total lines of code: {df['lines'].sum():,}")
print(f"Average file size: {df['size'].mean():.0f} bytes")
print(f"\nLargest files:")
print(df.nlargest(5, 'size')[['path', 'size', 'lines']])
```

### Parallel Processing Found Files

```python
import vexy_glob
from concurrent.futures import ProcessPoolExecutor
import os

def process_file(path):
    """Process a single file (e.g., count lines)"""
    try:
        with open(path, 'r', encoding='utf-8') as f:
            return path, sum(1 for _ in f)
    except:
        return path, 0

# Process all Python files in parallel
with ProcessPoolExecutor() as executor:
    # Get all files as a list
    files = vexy_glob.find("**/*.py", as_list=True)
    
    # Process in parallel
    results = executor.map(process_file, files)
    
    # Collect results
    total_lines = 0
    for path, lines in results:
        total_lines += lines
        if lines > 1000:
            print(f"Large file: {path} ({lines} lines)")
    
    print(f"\nTotal lines of code: {total_lines:,}")
```

## Migration Guide

### Migrating from `glob`

```python
# OLD: Using glob
import glob
import os

# Find all Python files
files = glob.glob("**/*.py", recursive=True)

# Filter by size manually
large_files = []
for f in files:
    if os.path.getsize(f) > 1024 * 1024:  # 1MB
        large_files.append(f)

# NEW: Using vexy_glob
import vexy_glob

# Find large Python files directly
large_files = vexy_glob.find("**/*.py", min_size=1024*1024, as_list=True)
```

### Migrating from `pathlib`

```python
# OLD: Using pathlib
from pathlib import Path

# Find all Python files
files = list(Path(".").rglob("*.py"))

# Filter by modification time manually
import datetime
recent = []
for f in files:
    if f.stat().st_mtime > (datetime.datetime.now() - datetime.timedelta(days=7)).timestamp():
        recent.append(f)

# NEW: Using vexy_glob
import vexy_glob

# Find recent Python files directly
recent = vexy_glob.find("**/*.py", mtime_after="-7d", as_path=True, as_list=True)
```

### Migrating from `os.walk`

```python
# OLD: Using os.walk
import os

# Find all .txt files
txt_files = []
for root, dirs, files in os.walk("."):
    for file in files:
        if file.endswith(".txt"):
            txt_files.append(os.path.join(root, file))

# NEW: Using vexy_glob
import vexy_glob

# Much simpler and faster!
txt_files = vexy_glob.find("**/*.txt", as_list=True)
```

## Development

This project is built with `maturin` - a tool for building and publishing Rust-based Python extensions.

### Prerequisites

- Python 3.8 or later
- Rust toolchain (install from [rustup.rs](https://rustup.rs/))
- `uv` for fast Python package management (optional but recommended)

### Setting Up Development Environment

```bash
# Clone the repository
git clone https://github.com/vexyart/vexy-glob.git
cd vexy-glob

# Set up a virtual environment (using uv for faster installation)
pip install uv
uv venv
source .venv/bin/activate  # On Windows: .venv\Scripts\activate

# Install development dependencies
uv sync

# Build the Rust extension in development mode
python sync_version.py  # Sync version from git tags to Cargo.toml
maturin develop

# Run tests
pytest tests/

# Run benchmarks
pytest tests/test_benchmarks.py -v --benchmark-only
```

### Building Release Artifacts

The project uses a streamlined build system with automatic versioning from git tags.

#### Quick Build

```bash
# Build both wheel and source distribution
./build.sh
```

This script will:
1. Sync the version from git tags to `Cargo.toml`
2. Build an optimized wheel for your platform
3. Build a source distribution (sdist)
4. Place all artifacts in the `dist/` directory

#### Manual Build

```bash
# Ensure you have the latest tags
git fetch --tags

# Sync version to Cargo.toml
python sync_version.py

# Build wheel (platform-specific)
python -m maturin build --release -o dist/

# Build source distribution
python -m maturin sdist -o dist/
```

### Build System Details

The project uses:
- **maturin** as the build backend for creating Python wheels from Rust code
- **setuptools-scm** for automatic versioning based on git tags
- **sync_version.py** to synchronize versions between git tags and `Cargo.toml`

Key files:
- `pyproject.toml` - Python project configuration with maturin as build backend
- `Cargo.toml` - Rust project configuration
- `sync_version.py` - Version synchronization script
- `build.sh` - Convenience build script

### Versioning

Versions are managed through git tags:

```bash
# Create a new version tag
git tag v1.0.4
git push origin v1.0.4

# Build with the new version
./build.sh
```

The version will be automatically detected and used for both the Python package and Rust crate.

### Project Structure

```
vexy-glob/
├── src/                    # Rust source code
│   ├── lib.rs             # Main Rust library with PyO3 bindings
│   └── ...
├── vexy_glob/             # Python package
│   ├── __init__.py        # Python API wrapper
│   ├── __main__.py        # CLI implementation
│   └── ...
├── tests/                 # Python tests
│   ├── test_*.py          # Unit and integration tests
│   └── test_benchmarks.py # Performance benchmarks
├── Cargo.toml             # Rust project configuration
├── pyproject.toml         # Python project configuration
├── sync_version.py        # Version synchronization script
└── build.sh               # Build automation script
```

### CI/CD

The project uses GitHub Actions for continuous integration:
- Testing on Linux, macOS, and Windows
- Python versions 3.8 through 3.12
- Automatic wheel building for releases
- Cross-platform compatibility testing

## Exceptions and Error Handling

### Exception Hierarchy

```python
VexyGlobError(Exception)
├── PatternError(VexyGlobError, ValueError)
│   └── Raised for invalid glob patterns
├── SearchError(VexyGlobError, IOError)  
│   └── Raised for I/O or permission errors
└── TraversalNotSupportedError(VexyGlobError, NotImplementedError)
    └── Raised for unsupported operations
```

### Error Handling Examples

```python
import vexy_glob
from vexy_glob import VexyGlobError, PatternError, SearchError

try:
    # Invalid pattern
    for path in vexy_glob.find("[invalid"):
        print(path)
except PatternError as e:
    print(f"Invalid pattern: {e}")

try:
    # Permission denied or I/O error
    for path in vexy_glob.find("**/*", root="/root"):
        print(path)
except SearchError as e:
    print(f"Search failed: {e}")

# Handle any vexy_glob error
try:
    results = vexy_glob.find("**/*.py", content="[invalid regex")
except VexyGlobError as e:
    print(f"Operation failed: {e}")
```

## Platform-Specific Considerations

### Windows

- Use forward slashes `/` in patterns (automatically converted)
- Hidden files: Files with hidden attribute are included with `hidden=True`
- Case sensitivity: Windows is case-insensitive by default

```python
# Windows-specific examples
import vexy_glob

# These are equivalent on Windows
vexy_glob.find("C:/Users/*/Documents/*.docx")
vexy_glob.find("C:\\Users\\*\\Documents\\*.docx")  # Also works

# Find hidden files on Windows
for path in vexy_glob.find("**/*", hidden=True):
    print(path)
```

### macOS

- `.DS_Store` files are excluded by default (via .gitignore)
- Case sensitivity depends on file system (usually case-insensitive)

```python
# macOS-specific examples
import vexy_glob

# Exclude .DS_Store and other macOS metadata
for path in vexy_glob.find("**/*", exclude=["**/.DS_Store", "**/.Spotlight-V100", "**/.Trashes"]):
    print(path)
```

### Linux

- Always case-sensitive
- Hidden files start with `.`
- Respects standard Unix permissions

```python
# Linux-specific examples
import vexy_glob

# Find files in home directory config
for path in vexy_glob.find("~/.config/**/*.conf", hidden=True):
    print(path)
```

## Troubleshooting

### Common Issues

#### 1. No results found

```python
# Check if you need hidden files
results = list(vexy_glob.find("*"))
if not results:
    # Try with hidden files
    results = list(vexy_glob.find("*", hidden=True))

# Check if .gitignore is excluding files
results = list(vexy_glob.find("**/*.py", ignore_git=True))
```

#### 2. Pattern not matching expected files

```python
# Debug pattern matching
import vexy_glob

# Too specific?
print(list(vexy_glob.find("src/lib/test.py")))  # Only exact match

# Use wildcards
print(list(vexy_glob.find("src/**/test.py")))   # Any depth
print(list(vexy_glob.find("src/*/test.py")))    # One level only
```

#### 3. Content search not finding matches

```python
# Check regex syntax
import vexy_glob

# Wrong: Python regex syntax
results = vexy_glob.find("**/*.py", content=r"import\s+{re,os}")

# Correct: Standard regex
results = vexy_glob.find("**/*.py", content=r"import\s+(re|os)")

# Case sensitivity
results = vexy_glob.find("**/*.py", content="TODO", case_sensitive=False)
```

#### 4. Performance issues

```python
# Optimize your search
import vexy_glob

# Slow: Searching everything
for path in vexy_glob.find("**/*.py", content="import"):
    print(path)

# Fast: Limit scope
for path in vexy_glob.find("src/**/*.py", content="import", max_depth=3):
    print(path)

# Use exclusions
for path in vexy_glob.find(
    "**/*.py",
    exclude=["**/node_modules/**", "**/.venv/**", "**/build/**"]
):
    print(path)
```

### Build Issues

If you encounter build issues:

1. **Rust not found**: Install Rust from [rustup.rs](https://rustup.rs/)
2. **maturin not found**: Run `pip install maturin`
3. **Version mismatch**: Run `python sync_version.py` to sync versions
4. **Import errors**: Ensure you've run `maturin develop` after changes
5. **Build fails**: Check that you have the latest Rust stable toolchain

### Debug Mode

```python
import vexy_glob
import logging

# Enable debug logging
logging.basicConfig(level=logging.DEBUG)

# This will show internal operations
for path in vexy_glob.find("**/*.py"):
    print(path)
```

## FAQ

**Q: Why is vexy_glob so much faster than glob?**

A: vexy_glob uses Rust's parallel directory traversal, releases Python's GIL, and streams results as they're found instead of collecting everything first.

**Q: Does vexy_glob follow symbolic links?**

A: By default, no. Use `follow_symlinks=True` to enable. Loop detection is built-in.

**Q: Can I use vexy_glob with async/await?**

A: Yes! Use it with asyncio.to_thread():
```python
import asyncio
import vexy_glob

async def find_files():
    return await asyncio.to_thread(
        vexy_glob.find, "**/*.py", as_list=True
    )
```

**Q: How do I search in multiple directories?**

A: Call find() multiple times or use a common parent:
```python
# Option 1: Multiple calls
results = []
for root in ["src", "tests", "docs"]:
    results.extend(vexy_glob.find("**/*.py", root=root, as_list=True))

# Option 2: Common parent with specific patterns
results = vexy_glob.find("{src,tests,docs}/**/*.py", as_list=True)
```

**Q: Is the content search as powerful as ripgrep?**

A: Yes! It uses the same grep-searcher crate that powers ripgrep, including SIMD optimizations.

### Advanced Configuration

#### Custom Ignore Files

```python
import vexy_glob

# By default, respects .gitignore
for path in vexy_glob.find("**/*.py"):
    print(path)

# Also respects .ignore and .fdignore files
# Create .ignore in your project root:
# echo "test_*.py" > .ignore

# Now test files will be excluded
for path in vexy_glob.find("**/*.py"):
    print(path)  # test_*.py files excluded
```

#### Thread Configuration

```python
import vexy_glob
import os

# Auto-detect (default)
for path in vexy_glob.find("**/*.py"):
    pass

# Limit threads for CPU-bound operations
for match in vexy_glob.find("**/*.py", content="TODO", threads=2):
    pass

# Max parallelism for I/O-bound operations
cpu_count = os.cpu_count() or 4
for path in vexy_glob.find("**/*", threads=cpu_count * 2):
    pass
```

### Contributing

We welcome contributions! Here's how to get started:

1. Fork the repository
2. Create a feature branch (`git checkout -b feature-name`)
3. Make your changes
4. Run tests (`pytest tests/`)
5. Format code (`cargo fmt` for Rust, `ruff format` for Python)
6. Commit with descriptive messages
7. Push and open a pull request

Before submitting:
- Ensure all tests pass
- Add tests for new functionality
- Update documentation as needed
- Follow existing code style

#### Running the Full Test Suite

```bash
# Python tests
pytest tests/ -v

# Python tests with coverage
pytest tests/ --cov=vexy_glob --cov-report=html

# Rust tests
cargo test

# Benchmarks
pytest tests/test_benchmarks.py -v --benchmark-only

# Linting
cargo clippy -- -D warnings
ruff check .
```

## API Stability and Versioning

vexy_glob follows [Semantic Versioning](https://semver.org/):

- **Major version (1.x.x)**: Breaking API changes
- **Minor version (x.1.x)**: New features, backwards compatible
- **Patch version (x.x.1)**: Bug fixes only

### Stable API Guarantees

The following are guaranteed stable in 1.x:

- `find()` function signature and basic parameters
- `glob()` and `iglob()` compatibility functions
- `SearchResult` object attributes
- Exception hierarchy
- CLI command structure

### Experimental Features

Features marked experimental may change:

- Thread count optimization algorithms
- Internal buffer size tuning
- Specific error message text

## Performance Tuning Guide

### For Maximum Speed

```python
import vexy_glob

# 1. Be specific with patterns
# Slow:
vexy_glob.find("**/*.py")
# Fast:
vexy_glob.find("src/**/*.py")

# 2. Use depth limits when possible
vexy_glob.find("**/*.py", max_depth=3)

# 3. Exclude unnecessary directories
vexy_glob.find(
    "**/*.py",
    exclude=["**/venv/**", "**/node_modules/**", "**/.git/**"]
)

# 4. Use file type filters
vexy_glob.find("**/*.py", file_type="f")  # Skip directories
```

### For Memory Efficiency

```python
# Stream results instead of collecting
# Memory efficient:
for path in vexy_glob.find("**/*"):
    process(path)  # Process one at a time

# Memor