mcp-local-vision 0.1.0
MCP Server that enables local agentic flows or inference to capture and to 'see/analyze' images in a continuous loop (for example, when using LMStudio for Agentic Flow and a VLM can retrieve but not actually see the real pixels of the picture and the flow breaks).
License: MIT
Requires Python: >=3.10

# 👁️ mcp-local-vision (Local Vision MCP Tool)

> **Give your local agent eyes. Close the autonomous loop.**

**mcp-local-vision** is the missing link for autonomous local AI. It allows standard MCP agents (Your local agent running on LMStudio/Ollama/vLLM, Claude Desktop, Goose, etc.) to **"see"** images, screenshots, and visual assets by routing them back through an call to the same inference engine and same model (in case the initial model was a VLM model of course so that it can see and reason about the image thus enhancing your workflows.
Most current MCP servers like Playwright, File Access and Screenshot tools are missing the ability to allow the Agentic AI mode to actually see and reason about images and just returning responses that the end user at the GUI level can see but the Agentic Model is blind to, thus hindering the agentic flowIt also supports cloud providers like Anthropic, Gemini, OpenAI, Openrouter etc...

## 🚫 The Problem: The "Blind Agent" Paradox

You have a cutting-edge local agent. You ask it to "Debug my UI."

1.  Agent launches the app.
2.  Agent takes a screenshot to check for errors.
3.  **Agent stops.** 🛑

Why? Because standard MCP tools only *save* screenshots to disk. The agent knows the file *exists*, but it cannot *see* the pixels. It is blind to its own actions.

## ✅ The Solution: The Visual Loop

**mcp-local-vision** bridges the file system and your inference engine.

1.  **Agent** takes a screenshot via Playwright/Puppeteer.
2.  **Agent** calls `analyze_image(path)`.
3.  **mcp-local-vision** instantly pipes the raw pixels to your already hosted model (no extra host, VRAM/RAM usag, just efficient routing/loop in case the initial model supports Vision, or choose any other endpoint where a VLM is hosted and it works too.
4.  **Agent** receives a detailed description (from it-self sometimes! for a full descroption on the image which can allow it to proceed further ("I see a 404 error on the dashboard").
5.  **Agent** takes the next action to fix it. **The loop continues.**

## 🚀 Features

*   **100% Local Privacy:** Your screenshots never leave your machine if you want to!  (when using `local hosted models such as LMStudio, llama.cpp, vllm, Ollama or others.)
*   **Multi-Provider Support:** Plug in **Anthropic (Claude)**, **Google Gemini**, **OpenRouter**, or **OpenAI** if you need cloud power.
*   **Smart Fallbacks:** Automatically handles "Reasoning" models (like DeepSeek-R1) and connection errors.
*   **Robust Validation:** Checks for valid image files before analysis.
*   **Zero-Config Install:** Run instantly with `uvx`.

## ⚡ Quick Start

### 1. Minimal Setup (Local)
No cloning required. If you use `uv`, you can run this instantly:

```bash
uvx mcp-local-vision
```

**Alpha quick start for LM Studio (local loop)**
1) Start LM Studio local inference server on `http://localhost:1234/v1` with a vision model (e.g., `llava`, `llama-3.2-vision`, or whichever you loaded).  
2) Set `LOCAL_MODEL` (optional; defaults to `llava`), plus any `LOCAL_INFERENCE_URL`/`LOCAL_INFERENCE_KEY` overrides.  
3) Run from source: `python src/mcp_local_vision/server.py` (ensure `PYTHONPATH=src`) or use the installed entrypoint: `uvx mcp-local-vision`.  
4) Point your MCP client to the command above; call `analyze_local_image("/abs/path/to/image.png")`.

### 2. Configuration (Claude Desktop / Agent Config)

Add this to your `claude_desktop_config.json` or `mcp_server_config.json`.

#### Method 1: Using `uvx` (Recommended)
This is the Python equivalent of `npx`. It automatically downloads and runs the latest version of the package in an isolated environment.

```json
{
  "mcpServers": {
    "local-vision": {
      "command": "uvx",
      "args": ["mcp-local-vision"],
      "env": {
        "LOCAL_INFERENCE_URL": "http://localhost:1234/v1"
      }
    }
  }
}
```

*Note: You can also use the explicit `uv` command if `uvx` is not in your PATH:*
```json
    "local-vision": {
      "command": "uv",
      "args": ["tool", "run", "mcp-local-vision"],
      ...
    }
```

#### Method 2: Manual / Development (Using local venv)
If you have cloned the repo and want to run it from source (like the example you provided):

```json
{
  "mcpServers": {
    "local-vision-dev": {
      "command": "/absolute/path/to/your/custom/venv/bin/python",
      "args": [
        "/absolute/path/to/mcp-local-vision/src/mcp_local_vision/server.py"
      ],
      "env": {
        "PYTHONPATH": "/absolute/path/to/mcp-local-vision/src",
        "LOCAL_INFERENCE_URL": "http://localhost:1234/v1"
      }
    }
  }
}
```
*Note: Replace `/absolute/path/to/...` with your actual paths.*

### 3. Backend Configuration

#### Option A: Local (LM Studio / vLLM) - *Default*
Set `LOCAL_INFERENCE_URL` in the `env` section (see above).

#### Option B: Ollama
```json
      "env": {
        "OLLAMA_URL": "http://localhost:11434/v1",
        "OLLAMA_MODEL": "llava"
      }
```
```
*And pass `provider='ollama'` when calling the tool.*

*Tip: If your agent is already using a Vision Logic Model (VLM) like `llama3.2-vision` or `qwen2.5-vl`, set `OLLAMA_MODEL` to that same model name to create an efficient self-loop without loading a second model.*

#### Option C: Cloud Providers (Anthropic / Google / OpenRouter)
```json
      "env": {
        "ANTHROPIC_API_KEY": "sk-ant-...",
        "GOOGLE_API_KEY": "AIza...",
        "OPENROUTER_API_KEY": "sk-or-..."
      }
```

## 🛠️ Usage

The server exposes one tool: `analyze_local_image`.

**Arguments:**
- `file_path` (string): Absolute path to the image file.
- `question` (string): What to ask the model (Default: "Describe this image in detail").
- `timeout` (int): Timeout in seconds (Default: 30).
- `provider` (string): One of `local` (default), `ollama`, `openai`, `anthropic`, `google`, `openrouter`.

**Example User Query:**
"Look at the screenshot at `/tmp/screen.png` using the `anthropic` provider and tell me if the tests passed."

## ❓ Troubleshooting

### The "Self-Call" Architecture
If you are using an agent (like a VLM in LM Studio) to use this tool, remember:
1.  The Agent sends a request to MCP.
2.  MCP reads the file.
3.  MCP sends the image BACK to LM Studio (or Ollama) for analysis.
4.  LM Studio returns the description.
5.  MCP returns the description to the Agent.

**Critical:** If you are using the LM Studio **GUI** to chat, you must ensure the **Local Inference Server** is also turned ON and running on port 1234 (or your configured port). The GUI itself does not expose the API unless the server is started.

### Common Errors
- **"Connection Refused"**: Check that your local server (LM Studio/Ollama) is actually running.
- **"Model returned empty response"**: You might be using a text-only model (like Llama 3) instead of a Vision model (like Qwen2-VL or LLaVA).