owlsight 2.7.0b1
Owlsight is a command-line tool combining open-source AI models with Python functionality to create a powerful AI assistant.
License: MIT
Requires Python: <3.13,>=3.9

# Owlsight

**Owlsight** is a command-line tool that combines Python programming with open-source language models.
It offers an interactive interface that allows you to execute Python code, shell commands, and use an AI assistant in one unified environment. 
Next to that, Owlsight offers an extensive set of tools in its backend-API, which enables you to use most of the existing CLI-functionaliy in your own Python scripts.

## Why owlsight?

Picture this: you are someone who dabbles in Python occasionally. Or you frequently use generative AI to accelerate your workflow, whether for generating code or working with data.
Often, this involves a tedious process—copying and pasting code between ChatGPT and your IDE, repeatedly switching contexts.

What if you could eliminate this friction?

Owlsight brings Python and generative AI together in an intuitive Command Line Interface, streamlining your workflow by integrating them into a single, unified platform. 
No more toggling between windows, no more manual code transfers. With Owlsight, you get the full power of Python and AI, all in one place—simplifying your process and boosting productivity. 
Owlsight has been designed to be a swiss-army knife for Python and AI with a core focus on open-source models, allowing you to execute code directly from model prompts and access this code directly from the Python interpreter.

## Installation of the CLI:

You can install Owlsight using pip:

```bash
pip install owlsight
```

### Installation Options and Dependencies

A basic installation includes only the core dependencies needed for the transformers library and basic functionality. For access to specific features, you will need to install optional dependency groups:

#### Optional Feature Modules

To add GGUF model support (using llama-cpp-python):
```
pip install owlsight[gguf]
```

To add ONNX model support (optimized model inference):
```
pip install owlsight[onnx]
```

To add multimodal functionality (image processing, OCR):
```
pip install owlsight[multimodal]
```

To add web search and scraping capabilities:
```
pip install owlsight[search]
```

To add voice control functionality:
```
pip install owlsight[voice]
```

For operating in an offline environment with tika-server.jar file, enabling you to use the `DocumentReader` class (which includes Apache Tika functionality):
```
pip install owlsight[offline]
```

#### Comprehensive Installation
To install all packages and features:
```
pip install owlsight[all]
```

### Available Features Based on Installation

| Feature | Basic Install | Required Extra |
|---------|---------------|----------------|
| Transformers models | ✓ | - |
| GGUF models | ✗ | [gguf] |
| ONNX models | ✗ | [onnx] |
| Image processing | ✗ | [multimodal] |
| Web search/scraping | ✗ | [search] |
| Voice control | ✗ | [voice] |
| Offline document reading (using Apache Tika) | ✗ | [offline] |
| Development | ✗ | [dev] |

### Security and Performance Notes

- Using multiple flags is a conscious design choice to give users more control over the behavior of the application and prevent "dependency hell".
- The application is designed to gracefully handle missing dependencies - you will receive helpful warning/error messages if you attempt to use a feature without the required dependencies.
- Some libraries like llama-cpp-python and pytorch may require specific configurations depending on your hardware.
- If you want most useful features out of the box, it is recommended to pip install Owlsight with the [all] option. This will install owlsight with the following flags: gguf, onnx, multimodal, search
- Recommended python versions: 3.10, 3.11, 3.12. Lower or higher python versions may not support all features, especially due to package incompatibilities.

## Usage

After installation, launch Owlsight in the terminal by running the following command:

```
owlsight
```

This will present you with some giant ASCII-art of an owl and information which tells you whether you have access to an active GPU (assuming you use CUDA).

Then, you are presented with the mainmenu:

```
Current choice:
> how can I assist you?
shell
python
config: main
save
load
clear history
quit
```

A choice can be made in the mainmenu by pressing the UP and DOWN arrow keys.
In the config menu, the LEFT and RIGHT arrow keys can be used to navigate between the different sections.
From the config sections, press "back" to go back to the mainmenu.
Press ENTER to select an option.
If you want to change an option, press ENTER to confirm the change.

### Keyboard Shortcuts

When working with the editable option, the following keyboard shortcuts are available:

- **Ctrl+A**: Select all text in the current editable field
- **Ctrl+C**: Copy selected text
- **Ctrl+Y**: Paste selected text

### Getting Started

Now, lets start out by loading a model. Go to **config** and toggle a few times to the right to reach the **huggingface** section. Choose a task like *text-generation* and press ENTER. 

Then, use the *search* option to search for a model. 
You can first type in keywords before searching, like "llama gguf". This will give you results from the Huggingface modelhub which are related to models in the llama-family in GGUf format.

Press ENTER to see the top_k results. Use the LEFT and RIGHT arrow keys in the *select_model* option to select a model and press ENTER to load it.

### Available Commands

The following available commands are available from the mainmenu:

* **How can I assist you**: Ask a question or give an instruction. By default, model responses are streamed to the console.
* **shell** : Execute shell commands. This can be useful for pip installing python libraries inside the application.
* **python** : Enter a Python interpreter. Press exit() to return to the mainmenu.
* **config: main** : Modify the *main*, *model* , *generate* or *rag* configuration settings.
* **save/load** : Save or load a configuration file.
* **clear history** : Clear the chat history and cache folder.
* **quit** : Exit the application.

### Voice Control

Owlsight supports voice control functionality when installed with `pip install owlsight[voice]`. This allows you to control the application using voice commands.

To enable voice control, use the `--voice` flag when starting Owlsight:
```bash
owlsight --voice
```

You can customize the voice control behavior using JSON-based configuration:

```bash
# Custom key mappings (spoken words to keyboard actions)
owlsight --voice --word-to-key '{
    "backward": "left",
    "forward": "right",
    "save": ["ctrl", "s"],
    "select all": ["ctrl", "a"]
}'

# Custom word substitutions
owlsight --voice --word-to-word '{
    "print": "print()",
    "function": "def my_function():",
    "exit": "exit()"
}'

# Advanced voice control settings
owlsight --voice --voicecontrol-kwargs '{
    "cmd_cooldown": 0.5,
    "debug": true,
    "language": "en",
    "model": "base.en",
    "key_press_interval": 0.1,
    "typing_interval": 0.05
}'
```

These options can be combined to create a fully customized voice control experience, which you can also utilize outside of the application.

### OpenAI-Compatible Server

Owlsight can run a local server that mimics the OpenAI Chat Completions API, allowing you to use OpenAI-compatible tools like Aider with your local models.

**Running the Server**

Start the server using the `owlsight-server` command, specifying a model and any model-specific parameters.

```bash
owlsight-server --model <model_identifier> [options]
```

-   `--model`: (Required) The model identifier (e.g., `gguf/MyModel`, `hf/mistralai/Mistral-7B`).
-   `--port`: Server port (default: `8000`).
-   `--host`: Server host (default: `127.0.0.1`).
-   Any other `--key value` arguments are passed directly to the model's processor.

**Example: Serving a GGUF Model**

Let's say you have downloaded `DeepSeek-R1-0528-Qwen3-8B-Q4_K_M.gguf` from Hugging Face:

```cmd
owlsight-server ^
  --model unsloth/DeepSeek-R1-0528-Qwen3-8B-GGUF ^
  --gguf__filename DeepSeek-R1-0528-Qwen3-8B-Q4_K_M.gguf ^
  --gguf__n_ctx 8192 ^
  --gguf__verbose true ^
  --gguf__n_gpu_layers -1 ^
  --port 8000
```

*   `--model`: Specifies the base model identifier known to Owlsight or a unique name you assign.
*   `--gguf__filename`: Tells the GGUF processor which specific `.gguf` file to load.
*   `--gguf__n_ctx`: Sets the context size for the GGUF model.
*   `--gguf__n_gpu_layers`: Offloads layers to GPU (-1 for all possible).
*   `--port`: Runs the server on port 8000.

**Testing and Integration**

*   **Test with Swagger UI**: Open `http://localhost:8000/docs` in your browser to send test requests to the `/v1/chat/completions` endpoint.

    **Example Request Body:**

    ```json
    {
        "model": "unsloth/DeepSeek-R1-0528-Qwen3-8B-GGUF",
        "messages": [
            {
                "role": "user",
                "content": "Tell me a joke about AI."
            }
        ],
        "stream": false,
        "max_tokens": 150,
        "temperature": 0.7
    }
    ```

    **Note:** The `model` field in the request *must* match the `--model` argument you used when starting the server, or at least the part after any prefix (e.g. if server started with `hf/org/model-name`, client can use `org/model-name` or `model-name`).

*   **List Models**: Query the `/v1/models` endpoint to see the active model.
*   **Integrate with Tools (e.g., Aider)**: Set environment variables to point your tool to the local server.

    ```cmd
    REM For Windows
    set OPENAI_API_BASE=http://localhost:8000/v1
    set OPENAI_API_KEY=dummy
    ```
    ```bash
    # For Linux/macOS
    export OPENAI_API_BASE="http://localhost:8000/v1"
    export OPENAI_API_KEY="dummy"
    ```
    Then, run your tool specifying the model name (e.g., `aider --model unsloth/DeepSeek-R1-0528-Qwen3-8B-GGUF`).

**Important Notes**

*   A server process serves only **one model** at a time.
*   The `usage` field (token counts) in API responses is currently populated with placeholders (0).

### Example Workflow

You can combine Python variables defined in the Python Interpreter together with language models in Owlsight through special double curly-brackets syntax.
For example:

```
python > a = 42
How can I assist you? > How much is {{a}} * 5?
```

```
answer -> 210
```

Additionally, you can also ask a model to write pythoncode and access that in the python interpreter.

From a model response, all generated python code will be extracted and can be edited or executed afterwards. This choice is always optional. After execution, the defined objects will be saved in the global namespace of the python interpreter for the remainder of the current active session. This is a powerful feature, which allows build-as-you-go for a wide range of complex tasks.

Example:

```
How can I assist you? > Can you write a function which reads an Excel file?
```

-> *model writes a function called read_excel*

```
python > excel_data = read_excel("path/to/excel")
```

### MultiModal Support

In Owlsight 2, special multimodal support is available for certain models that require additional input, like images or audio. In the backend, this is made possible with the **MultiModalProcessorTransformers** class. 
In the CLI, this can be done by setting the *config.model.model_id* to a multimodal model from the Huggingface modelhub. 
Keep in mind that this model should be a Pytorch model (so not GGUF or ONNX).
For convenience, it is recommended to select a model through the new Huggingface API in the configuration-settings (read below for more information).

The following tasks are supported for multimodal models:

- image-to-text
- automatic-speech-recognition
- visual-question-answering
- document-question-answering

These models require additional input, which can be passed in the prompt. 
The syntax for passing mediatypes can be done through special double-square brackets syntax, like so:

*How can I assist you?*
```text
[[image:path/to/file.jpg]]
```

The supported mediatypes are: *image*, *audio*.
For example, to pass an image to a document-question-answering model, you can use the following syntax:

*How can I assist you?*
```text
What is the first sentence in this image? [[image:path/to/image.jpg]]
```

## Python interpreter

Next to the fact that objects generated by model-generated code can be accessed, the Python interpreter also has some useful default functions, starting with the "owl_" suffix. These serve as utilityfunctions.

These are:
- `owl_import`: Import Python file to current namespace
- `owl_read`: Read file content from any supported format
- `owl_edit`: Edit file content
- `owl_terminal`: Execute shell commands. Useful for tool usage by an agent
- `owl_scrape`: Scrape urls
- `owl_show`: Display active objects in the Python namespace
- `owl_write`: Write content to text file
- `owl_history`: Display model chat history
- `owl_models`: Display loaded HuggingFace models in cache directory
- `owl_press`: Press keys for automation tasks
- `owl_save_namespace`: Save namespace to .dill file
- `owl_load_namespace`: Load namespace from .dill file
- `owl_tools`: Show available functions for tool calling
- `owl_search`: Search and get results from the web using DuckDuckGo's API
- `owl_search_and_scrape`: Search and scrape the web using DuckDuckGo's API. Uses both the `owl_search` and `owl_scrape` functions combined.
- `owl_create_document_searcher`: Create a DocumentSearcher instance with a given set of documents and a text splitter. This class is great for usage in a RAG scenario.

## Configurations

Owlsight uses a configuration file in JSON-format to adjust various parameters. The configuration is divided into five main sections: `main`, `model`,  `generate`, `rag` and `huggingface`. Here's an overview of the application architecture:

Main Menu:
- assistant: Chat with the loaded model. Use {{expression}} to pass python code directly. Or e.g. [[image: path/to/image.jpg]] to pass an image to the model
- shell: Execute shell commands
- python: Enter Python interpreter
- config: Configuration settings
  - main settings:
    - back: Return to previous menu
    - max_retries_on_error: Maximum number of retries for Python code error recovery. This parameter is only used when `prompt_retry_on_error` is set to True., Options: 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, Type: OptionType.TOGGLE
    - prompt_retry_on_error: Whether to prompt before retrying on error. Set this to True to avoid direct Python code execution on error!, Options: False, True, Type: OptionType.TOGGLE
    - prompt_code_execution: Whether to prompt before executing code. Set this to True to avoid direct Python code execution!, Options: False, True, Type: OptionType.TOGGLE
    - track_model_usage: Show metrics after a model response. Tracks GPU/CPU usage, amount of generated words and responsetime of model. NOTE: GPU tracking only works for PyTorch models., Options: False, True, Type: OptionType.TOGGLE
    - extra_index_url: Additional URL for Python package installation. Useful for example when installing python packages (through pip) from private repositories, Type: OptionType.EDITABLE
    - python_compile_mode: Compile mode in the Python Interpreter (main menu): 'exec' is suited for defining code blocks, 'single' for direct execution, Options: exec, single, Type: OptionType.TOGGLE
    - dynamic_system_prompt: Experimental feature: The model will first act as Prompt Engineer to create a new system prompt based on user input., Options: False, True, Type: OptionType.TOGGLE
    - default_config_on_startup: Link to a configuration file that will be loaded on startup., Type: OptionType.EDITABLE
    - sequence_on_loading: A list of key sequences to execute when loading the configuration. Uses owl_press functionality., Type: OptionType.EDITABLE
  - model settings:
    - back: Return to previous menu
    - model_id: Model identifier or path. The most important parameter in the configuration, as this will load the model to be used, Type: OptionType.EDITABLE
    - apply_chat_history: Toggle the inclusion of saved chat history in the prompt. Enable for chat models, disable for instruct models., Options: False, True, Type: OptionType.TOGGLE
    - system_prompt: System prompt defining model behavior, Type: OptionType.EDITABLE
    - model_kwargs: Additional parameters passed during model initialization. For llama-cpp, these get passed to llama_cpp.Llama. For transformers, these get passed to transformers.pipeline, Type: OptionType.EDITABLE
    - transformers__device: Device for transformers model, Options: None, cpu, cuda, mps, Type: OptionType.TOGGLE
    - transformers__quantization_bits: Quantization bits for transformers model, Options: None, 4, 8, 16, Type: OptionType.TOGGLE
    - transformers__stream: Whether to stream input to transformers model, Options: False, True, Type: OptionType.TOGGLE
    - gguf__filename: GGUF model filename, Type: OptionType.EDITABLE
    - gguf__verbose: Verbose output for GGUF model, Options: False, True, Type: OptionType.TOGGLE
    - gguf__n_ctx: Context length for GGUF model, Options: 32, 64, 128, 256, 512, 1024, 2048, 4096, 8192, 16384, 32768, 65536, 131072, 262144, 524288, Type: OptionType.TOGGLE
    - gguf__n_gpu_layers: Number of layers from the model which are offloaded to the GPU, Options: -1, 0, 1, 2, 4, 8, 16, 32, 64, 128, 256, 512, 1024, Type: OptionType.TOGGLE
    - gguf__n_batch: Batch size to be used by GGUF model, Options: 4, 8, 16, 32, 64, 128, 256, 512, 1024, 2048, 4096, 8192, 16384, Type: OptionType.TOGGLE
    - gguf__n_cpu_threads: Number of CPU threads to be used by GGUF model., Options: 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, Type: OptionType.TOGGLE
    - onnx__model_dir: Directory containing local ONNX model, Type: OptionType.EDITABLE
    - onnx__verbose: Verbose output for ONNX model, Options: False, True, Type: OptionType.TOGGLE
    - onnx__n_cpu_threads: Number of CPU threads to be used by ONNX model, Options: 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, Type: OptionType.TOGGLE
  - generate settings:
    - back: Return to previous menu
    - stop_words: stop_words that stop text generation. This can be useful for getting more control over when modelgeneration should stop. Pass these like `['stop', 'word']`, Type: OptionType.EDITABLE
    - max_new_tokens: Maximum amount of tokens to generate, Options: 32, 64, 128, 256, 512, 1024, 2048, 4096, 8192, 16384, 32768, 65536, 131072, 262144, 524288, Type: OptionType.TOGGLE
    - temperature: Temperature for model generation, Options: 0.0, 0.05, 0.1, 0.15, 0.2, 0.25, 0.3, 0.35, 0.4, 0.45, 0.5, 0.55, 0.6, 0.65, 0.7, 0.75, 0.8, 0.85, 0.9, 0.95, 1.0, Type: OptionType.TOGGLE
    - generation_kwargs: Additional generation parameters, like top_k, top_p, etc. Pass these like `{'top_k': 4, 'top_p': 0.9}`, Type: OptionType.EDITABLE
  - rag settings:
    - back: Return to previous menu
    - active: Whether RAG for python libraries is active. If True, the search-results will be implicitly added as context to the modelprompt and when pressing ENTER, search-results will be shown, Options: False, True, Type: OptionType.TOGGLE
    - target_library: Target python library for to use for RAG. If the library is not installed in the active environment, a warning will be showed with available options, Type: OptionType.EDITABLE
    - top_k: Number of most matching RAG results to return, based on `search` query, Options: 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, Type: OptionType.TOGGLE
    - sentence_transformer_weight: Weight for the embedding model. TFIDF-weight is 1 - `sentence_transformer_weight`, Options: 0.0, 0.05, 0.1, 0.15, 0.2, 0.25, 0.3, 0.35, 0.4, 0.45, 0.5, 0.55, 0.6, 0.65, 0.7, 0.75, 0.8, 0.85, 0.9, 0.95, 1.0, Type: OptionType.TOGGLE
    - sentence_transformer_name_or_path: Name or path to a sentence-transformer model, which is used for embedding, Type: OptionType.EDITABLE
    - search: RAG search query. Press ENTER to show the `top_k` results. Only used when `active` is True, Type: OptionType.EDITABLE
  - agentic settings:
    - back: Return to previous menu
    - active: Toggle whether the agentic system is active. Available tools concerns an existing subset of functions (and every new defined one) in the Python Interpreter namespace., Options: False, True, Type: OptionType.TOGGLE
    - additional_information: Additional information specifically for the Tool agent. E.g. 'Do NOT use owl_scrape and owl_search, because there is no internet connection', Type: OptionType.EDITABLE
    - exclude_tools: Comma-separated list of tools (as string) to exclude from the available tools. These tools can be used by the Tool agent. E.g. ['owl_scrape,owl_search'], Type: OptionType.EDITABLE
    - config_per_agent: Set configurations per agent, allowing unique models for each agent type. For Example: {'PlanAgent': 'path/to/config.json', 'PlanValidationAgent': 'path/to/config.json', 'ToolCreationAgent': 'path/to/config.json', 'ToolSelectionAgent': 'path/to/config.json', 'ObservationAgent': 'path/to/config.json', 'FinalAgent': 'path/to/config.json'}, Type: OptionType.EDITABLE
  - huggingface settings:
    - back: Return to previous menu
    - search: Search for a model on the Hugging Face Hub by pressing ENTER. Keywords can be used optionally to finetune searchresults, e.g. 'llama 3b gguf', Type: OptionType.EDITABLE
    - top_k: Top number of Hugging Face results to return. The results will be sorted by highest score first, Options: 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, Type: OptionType.TOGGLE
    - select_model: Select and load a model from the Hugging Face Hub by toggling through the options found by `search`, Type: OptionType.TOGGLE
    - task: Filter Hugging Face models by task. When using `search`, the results will be filtered directly by chosen task, Options: None, text-generation, text2text-generation, translation, summarization, image-to-text, automatic-speech-recognition, visual-question-answering, document-question-answering, Type: OptionType.TOGGLE
- save: Save current configuration as JSON-file
- load: Load a configuration from a JSON-file
- clear history: Clear owlsight cache (directory called '.owlsight') and chat history
- quit: Exit application

Here is an example of what the default configuration looks like:

```json
{
    "main": {
        "max_retries_on_error": 3,
        "prompt_retry_on_error": true,
        "prompt_code_execution": true,
        "track_model_usage": false,
        "extra_index_url": "",
        "python_compile_mode": "single",
        "dynamic_system_prompt": false,
        "default_config_on_startup": "",
        "sequence_on_loading": []
    },
    "model": {
        "model_id": "",
        "apply_chat_history": true,
        "system_prompt": "",
        "model_kwargs": {},
        "transformers__device": null,
        "transformers__quantization_bits": null,
        "transformers__stream": true,
        "gguf__filename": "",
        "gguf__verbose": false,
        "gguf__n_ctx": 2048,
        "gguf__n_gpu_layers": 0,
        "gguf__n_batch": 8,
        "gguf__n_cpu_threads": 8,
        "onnx__model_dir": "",
        "onnx__verbose": false,
        "onnx__n_cpu_threads": 8
    },
    "generate": {
        "stop_words": [],
        "max_new_tokens": 2048,
        "temperature": 0.7,
        "generation_kwargs": {}
    },
    "rag": {
        "active": false,
        "target_library": "",
        "top_k": 10,
        "sentence_transformer_weight": 0.0,
        "sentence_transformer_name_or_path": "Alibaba-NLP/gte-base-en-v1.5",
        "search": ""
    },
    "agentic": {
        "active": false,
        "additional_information": "",
        "exclude_tools": [],
        "config_per_agent": {}
    },
    "huggingface": {
        "search": "",
        "top_k": 10,
        "select_model": "",
        "task": null
    }
}
```

Configuration files can be saved (`save`) and loaded (`load`) through the main menu.

### Changing configurations

To update a configuration, simply modify the desired value and press **ENTER** to confirm the change. Please note that only one configuration setting can be updated at a time, and the change will only go into effect once **ENTER** has been pressed.

## Temporary environment

During an Owlsight session, a temporary environment is created within the homedirectory, called ".owlsight_packages". Newly installed python packages will be installed here. This folder will be removed if the session ends. If you want to persist installed packages, simply install them outside of Owlsight.

## Error Handling and Auto-Fix

Owlsight automatically tries to fix and retry any code that encounters a **ModuleNotFoundError** by installing the required package and re-executing the code. It can also attempt to fix errors in its own generated code. This feature can be controlled by the *max_retries_on_error* parameter in the configuration file.

## Agentic system

Owlsight implements a multistep agentic system, which allows for more complex tasks to be executed than would normally be possible with one language model.
This agentic system is accessible through the CLI by setting the *config.agentic.active* parameter to *true*.

The agents consist of:
['PlanAgent', 'PlanValidationAgent', 'ToolCreationAgent', 'ToolSelectionAgent', 'ObservationAgent', 'FinalAgent']

First, an Executionplan is created by the PlanAgent. 
This plan contain several steps, where each step is assigned to a downstream agent.

To make sure the plan is valid, the plan is validated by the PlanValidationAgent.

The mainagents for executing steps from the executionplan are ToolSelectionAgent and ToolCreationAgent.

ToolSelectionAgent is the main agent that is used to select and run tools.
The following tools from the Python interpreter are available out of the box for ToolSelectionAgent to use:
owl_read, owl_write, owl_edit, owl_search, owl_scrape, owl_terminal

After every ToolSelectionAgent step, the ObservationAgent is used to summarize the result of the tool execution.
This so that the information provided by ToolSelectionAgent is shorter, richer and free of noise.
This makes the information better suited for downstream agents.

ToolCreationAgent is the main agent that is used to create new tools.
Using this agent, a new tool can be created dynamicly in Python and added to the AVAILABLE TOOLS registry.
This tool can then later be used by ToolSelectionAgent.

The final agent is the FinalAgent, which is used to provide the final response to the user based on all previous steps. 

Here is a diagram illustrating the agentic flow within Owlsight-CLI (config:agentic):

<img src="docs/agent_flow_diagram.png" alt="Agent Flow Diagram" width="600"/>

## API Examples

Owlsight can also be used as a library in Python scripts. The main classes are the `TextGenerationProcessor` family, which can be imported from the `owlsight` package. 

Here is a simple example of how to use it:
```python
from owlsight import TextGenerationProcessorGGUF
# If you want to use another type of text-generation model, you can import the other classes: TextGenerationProcessorONNX, TextGenerationProcessorTransformers

processor = TextGenerationProcessorGGUF(
    model_id=r"path	o\Phi-3-mini-128k-instruct.Q5_K_S.gguf",
)

question = "What is the meaning of life?"

for token in processor.generate_stream(question):
    print(token, end="", flush=True)
```

Alternatively, there is a lot more to explore in the `owlsight` package.
Here is an example on how to use the `DocumentSearcher` class for simple document retrieval:
```python
from owlsight import DocumentSearcher, SentenceTextSplitter, SemanticTextSplitter

docs = {
    "doc1": "Quantum mechanics describes nature at atomic scales, introducing wave-particle duality and entanglement.",
    "doc2": "General relativity redefines gravity as spacetime curvature, predicting black holes and gravitational waves.",
    "doc3": "Quantum gravity aims to unify quantum mechanics and relativity, with theories like string theory and LQG.",
    "doc4": "String theory is a framework for understanding the universe, with models like the Minkowski space-time and the Einstein-Hilbert action.",
    "doc5": "LQG is a framework for quantum gravity, with models like the Einstein action and the black hole metric."
}

# Experiment with different text splitters
# splitter = SemanticTextSplitter()
splitter = SentenceTextSplitter(n_sentences=2)

searcher = DocumentSearcher(
    documents=docs,
    text_splitter=splitter,
    cache_dir="quantum_gravity",
    cache_dir_suffix="test",
)

query = "black holes in quantum gravity"
results = searcher.search(query, top_k=2)
```

Or a more advanced example of similarity search, where some websites are scraped and being splitted in chunks based on their semantic similarity.
```python
from owlsight import OwlDefaultFunctions, SemanticTextSplitter, DocumentSearcher

if __name__ == "__main__":
    owl_funcs = OwlDefaultFunctions({})

    # List of AI/ML related URLs to scrape
    urls = [
        "https://plato.stanford.edu/entries/artificial-intelligence/",  # Stanford's AI Philosophy
        "https://www.nature.com/articles/s42256-019-0088-2",  # Nature's Deep Learning overview
    ]

    scraped_text = owl_funcs.owl_scrape(urls)
    model_name = "sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2"
    doc_splitter = SemanticTextSplitter(percentile=0.99, target_chunk_length=400, model_name=model_name)
    doc_searcher = DocumentSearcher(scraped_text, sentence_transformer_model=model_name, text_splitter=doc_splitter)
    df = doc_searcher.search("reinforcement learning")
    seperator = "-" * 100
    for idx, row in enumerate(df.iterrows(), start=1):
        print(seperator)
        score = row[1]["aggregated_score"]
        print(f"Rank: {idx} (Score: {score:.2f})")
        print(f"Document name: {row[1]['document_name']}")
        print(row[1]["document"])

```

## API Documentation

The following section details all the objects and functions available in the Owlsight API:


### Classes

#### TextGenerationProcessorOnnx

```python
class TextGenerationProcessorOnnx(model_id: str, onnx__verbose: bool = False, onnx__n_cpu_threads: int = 8, onnx__model_dir: Optional[str] = None, token: Optional[str] = None, apply_chat_history: bool = False, system_prompt: Optional[str] = None, model_kwargs: Optional[dict] = None, apply_tools: Optional[List[dict]] = None, **kwargs: Any) -> None
```

Text generation processor using ONNX Runtime optimized models.

This processor enables text generation using ONNX-optimized models,
which can run on both CPU and GPU. Supports both local models and models from
Hugging Face Hub.

Parameters
----------
model_id : str
    Path to local ONNX model or Hugging Face model ID
onnx__verbose : bool, default=False
    Enable verbose ONNX Runtime logging
onnx__n_cpu_threads : int, default=8
    Number of CPU threads for computation
onnx__model_dir : str, optional
    Specific model directory when multiple valid ones exist
token : str, optional
    Hugging Face token for private models
apply_chat_history : bool, default=False
    Whether to maintain conversation history
system_prompt : str, optional
    System prompt prepended to all inputs
model_kwargs : dict, optional
    Additional keyword arguments to pass to the model.
    Default is None.

Notes
-----
- ONNX models typically offer better CPU performance than PyTorch
- Thread count affects CPU performance significantly
- Models must be ONNX-optimized versions of transformers models

**Examples:**

```python
--------
>>> # Load local ONNX model
>>> processor = TextGenerationProcessorOnnx("path/to/model")
>>>
>>> # Load from Hugging Face
>>> processor = TextGenerationProcessorOnnx(
...     "onnx-community/Llama-2-7B-Instruct-ONNX",
...     onnx__n_cpu_threads=12
... )
```

**Methods:**

- `apply_chat_template(self, input_data: str, tokenizer: transformers.tokenization_utils.PreTrainedTokenizer) -> str`
  - Apply chat template to the input text.
- `clear_history(self) -> None`
  - Clear the chat history.
- `generate(self, input_data: str, max_new_tokens: int = 512, temperature: float = 0.0, stop_words: Optional[List[str]] = None, buffer_wordsize: int = 10, generation_kwargs: Optional[Dict[str, Any]] = None) -> str`
  - Generate text response for the given input.
- `generate_openai_comp(self, messages: List[Dict[str, str]], **openai_kwargs: Any) -> Union[Dict[str, Any], Generator[Dict[str, Any], NoneType, NoneType]]`
  - OpenAI-compatible wrapper around ``generate`` / ``generate_stream``.
- `generate_stream(self, input_data: str, max_new_tokens: int = 512, temperature: float = 0.0, stop_words: Optional[List[str]] = None, generation_kwargs: Optional[Dict[str, Any]] = None)`
  - Stream generated text tokens one by one.
- `get_history(self) -> List[Dict[str, str]]`
  - Get the chat history.
- `get_max_context_length(self) -> Optional[int]`
  - Get maximum context length for the model.
- `list_valid_repo_files(repo_id: str) -> List[str]`
- `pre_validate_model_id(model_id: str, onnx__model_dir: str)`
  - Validate the model_id and model_directory before using `snapshot_download`.
- `update_history(self, input_data: str, generated_text: str) -> None`
  - Update the chat history with the input and generated text.

#### TextGenerationProcessorTransformers

```python
class TextGenerationProcessorTransformers(model_id: str, transformers__device: Optional[str] = None, transformers__quantization_bits: Optional[int] = None, transformers__stream: bool = True, bnb_kwargs: Optional[dict] = None, tokenizer_kwargs: Optional[dict] = None, task: Optional[str] = None, apply_chat_history: bool = False, system_prompt: str = '', apply_tools: Optional[List[dict]] = None, model_kwargs: Optional[dict] = None, **kwargs)
```

Text generation processor using transformers library.

**Methods:**

- `apply_chat_template(self, input_data: str, tokenizer: transformers.tokenization_utils.PreTrainedTokenizer) -> str`
  - Apply chat template to the input text.
- `clear_history(self) -> None`
  - Clear the chat history.
- `generate(self, input_data: str, max_new_tokens: int = 512, temperature: float = 0.0, stop_words: Optional[List[str]] = None, generation_kwargs: Optional[Dict[str, Any]] = None) -> str`
  - Generate text response.
- `generate_openai_comp(self, messages: List[Dict[str, str]], **openai_kwargs: Any) -> Union[Dict[str, Any], Generator[Dict[str, Any], NoneType, NoneType]]`
  - OpenAI-compatible wrapper around ``generate`` / ``generate_stream``.
- `generate_stream(self, input_data: str, max_new_tokens: int = 512, temperature: float = 0.0, stop_words: Optional[List[str]] = None, generation_kwargs: Optional[Dict[str, Any]] = None) -> Generator[str, NoneType, NoneType]`
  - Generate streaming text response.
- `get_history(self) -> List[Dict[str, str]]`
  - Get the chat history.
- `get_max_context_length(self) -> Optional[int]`
  - Get the maximum context length for the model.
- `pipe_call(self, input_data: Union[str, List[str]], **gen_kwargs) -> Any`
  - Call the pipeline with input data and kwargs, supporting batch processing.
- `prepare_generation(self, input_data: str, max_new_tokens: int, temperature: float, stop_words: Optional[List[str]], generation_kwargs: Optional[Dict[str, Any]], streaming: bool = False, apply_chat_template: bool = True) -> Tuple[str, Dict[str, Any]]`
  - Prepare generation parameters.
- `update_history(self, input_data: str, generated_text: str) -> None`
  - Update the chat history with the input and generated text.

#### TextGenerationProcessorGGUF

```python
class TextGenerationProcessorGGUF(model_id: str, gguf__filename: str = '', gguf__verbose: bool = False, gguf__n_ctx: Optional[int] = None, gguf__n_gpu_layers: int = 0, gguf__n_batch: Optional[int] = None, gguf__n_cpu_threads: Optional[int] = None, apply_chat_history: bool = False, system_prompt: str = '', model_kwargs: Dict[str, Any] = None, apply_tools: Optional[List[dict]] = None, **kwargs)
```

Text generation processor for GGUF models using llama-cpp.

This processor enables efficient text generation using GGUF-quantized models,
which are optimized for CPU and GPU inference. Supports both local models and
models from Hugging Face Hub.

Parameters
----------
model_id : str
    Path to local GGUF model or Hugging Face model ID
gguf__filename : str, optional
    Specific GGUF file to load when using Hugging Face model ID
gguf__verbose : bool, default=False
    Enable verbose logging from llama-cpp
gguf__n_ctx : int, optional
    Context window size. Larger values allow longer conversations but use more memory
gguf__n_gpu_layers : int, default=0
    Number of layers to offload to GPU. Set >0 for GPU acceleration
gguf__n_batch : int, optional
    Batch size for generation. Increase for faster generation, at the cost of memory.
gguf__n_cpu_threads : int, optional
    The number of CPU threads to use for generation. Increase for much faster generation if multiple cores are available.
apply_chat_history : bool, default=False
    Whether to maintain conversation history
system_prompt : str, default=""
    System prompt prepended to all inputs
model_kwargs : Optional[Dict[str, Any]]
    Additional arguments passed for the model.
    These get passed to `transformers.pipeline` function as `model_kwargs` argument.
    Default is None.

Notes
-----
- GPU acceleration requires llama-cpp-python build specifically with CUDA support
- Context size (n_ctx) affects memory usage significantly
- For optimal performance, adjust n_batch and n_cpu_threads based on hardware

**Examples:**

```python
--------
>>> # Load local GGUF model
>>> processor = TextGenerationProcessorGGUF("path/to/model.gguf", gguf__n_gpu_layers=20)
>>>
>>> # Load from Hugging Face with GPU
>>> processor = TextGenerationProcessorGGUF(
...     "TheBloke/Llama-2-7B-GGUF",
...     gguf__filename="llama-2-7b.Q4_K_M.gguf",
...     gguf__n_gpu_layers=32
... )
```

**Methods:**

- `apply_chat_template(self, input_data: str) -> List[Dict[str, str]]`
  - Apply chat template to the input text.
- `clear_history(self) -> None`
  - Clear the chat history.
- `generate(self, input_data: str, max_new_tokens: int = 512, temperature: float = 0.1, stop_words: Optional[List[str]] = None, generation_kwargs: Optional[Dict[str, Any]] = None) -> str`
  - Generate text response for the given input.
- `generate_openai_comp(self, messages: List[Dict[str, str]], **openai_kwargs: Any) -> Union[Dict[str, Any], Generator[Dict[str, Any], NoneType, NoneType]]`
  - OpenAI-compatible wrapper around ``generate`` / ``generate_stream``.
- `generate_stream(self, input_data: str, max_new_tokens: int = 512, temperature: float = 0.1, stop_words: Optional[List[str]] = None, generation_kwargs: Optional[Dict[str, Any]] = None) -> Generator[str, NoneType, NoneType]`
  - Stream generated text token