semvec/pypi-packages huggingface dataset:
llama-index-readers-steamship 0.4.1
llama-index readers steamship integration
Requires Python: <4.0,>=3.9
Keywords: steamship

# LlamaIndex Readers Integration: SteamshipFile

## Overview

The SteamshipFile Reader allows you to load documents from persistent Steamship Files. Steamship is a platform for storing and managing files with advanced tagging capabilities.

For more detailed information about the SteamshipFile Reader, visit [SteamShip](steamship.com).

### Installation

You can install the SteamshipFile Reader via pip:

```bash
pip install llama-index-readers-steamship
```

This reader requires steamship API key, which can be acquired from [SteamShip](steamship.com).

### Usage

```python
from llama_index.readers.steamship import SteamshipFileReader

# Initialize SteamshipFileReader
reader = SteamshipFileReader(api_key="<Steamship API Key>")

# Load data from persistent Steamship Files
documents = reader.load_data(
    workspace="<Steamship Workspace>",
    query="<Steamship Tag Query>",
    file_handles=["smooth-valley-9kbdr"],
    collapse_blocks=True,
    join_str="\n\n",
)
```

This loader is designed to be used as a way to load data into
[LlamaIndex](https://github.com/run-llama/llama_index/tree/main/llama_index) and/or subsequently
used as a Tool in a [LangChain](https://github.com/hwchase17/langchain) Agent.
llama-index-llms-lmstudio 0.4.1
llama-index llms lmstudio integration
Requires Python: <4.0,>=3.9

# LlamaIndex Llms Integration: Lmstudio

```bash
pip install llama-index-llms-lmstudio
```

## Usage Steps

1. Open LM Studio App and go to the Local Server Tab
2. In the Configuration settings, enable Apply Prompt Formatting
3. Load the model of your choice
4. Start your server

```python
from llama_index.llms.lmstudio import LMStudio

llm = LMStudio(
    model_name="Hermes-2-Pro-Llama-3-8B",
    base_url="http://localhost:1234/v1",
    temperature=0.7,
)

messages = [
    ChatMessage(
        role=MessageRole.SYSTEM,
        content="You an expert AI assistant. Help User with their queries.",
    ),
    ChatMessage(
        role=MessageRole.USER,
        content="What is the significance of the number 42?",
    ),
]

response = llm.chat(messages=messages)
print(str(response))
```
llama-index-packs-gmail-openai-agent 0.4.1
llama-index packs gmail_openai_agent integration
Requires Python: <4.0,>=3.9
Keywords: math, research, science

# Gmail OpenAI Agent Pack

Create an OpenAI agent pre-loaded with a tool to interact with Gmail. The tool used is the [Gmail LlamaHub tool](https://llamahub.ai/l/tools-gmail).

## CLI Usage

You can download llamapacks directly using `llamaindex-cli`, which comes installed with the `llama-index` python package:

```bash
llamaindex-cli download-llamapack GmailOpenAIAgentPack --download-dir ./gmail_pack
```

You can then inspect the files at `./gmail_pack` and use them as a template for your own project!

## Code Usage

You can download the pack to a the `./gmail_pack` directory:

```python
from llama_index.core.llama_pack import download_llama_pack

# download and install dependencies
GmailOpenAIAgentPack = download_llama_pack(
    "GmailOpenAIAgentPack", "./gmail_pack"
)

gmail_agent_pack = GmailOpenAIAgentPack()
```

From here, you can use the pack, or inspect and modify the pack in `./gmail_pack`.

The `run()` function is a light wrapper around `agent.chat()`.

```python
response = gmail_agent_pack.run("What is my most recent email?")
```

You can also use modules individually.

```python
# Use the agent
agent = gmail_agent_pack.agent
response = agent.chat("What is my most recent email?")

# Use the tool spec in another agent
from llama_index.core.agent.workflow import ReActAgent

tool_spec = gmail_agent_pack.tool_spec
agent = ReActAgent(tools=tool_spec.to_tool_lost(), llm=llm)
```
llama-index-storage-index-store-azure 0.5.1
llama-index index_store azure integration
Requires Python: <4.0,>=3.9

# LlamaIndex Index_Store Integration: Azure Table Storage

`AzureIndexStore` utilizes Azure Table Storage and Cosmos DB to provide an index storage solution for indexing documents or data.

## Installation

Before using the `AzureIndexStore`, ensure you have Python installed and then proceed to install the required packages:

```bash
pip install llama-index-storage-index-store-azure
pip install azure-data-tables
pip install azure-identity  # Only needed for AAD token authentication
```

## Initializing `AzureIndexStore`

`AzureIndexStore` can be initialized in several ways depending on the authentication method and the Azure service (Table Storage or Cosmos DB) you are using:

### 1. Using a Connection String

```py
from llama_index.storage.index_store.azure import AzureIndexStore
from llama_index.storage.kvstore.azure.base import ServiceMode

store = AzureIndexStore.from_connection_string(
    connection_string="your_connection_string_here",
    namespace="your_namespace",
    service_mode=ServiceMode.STORAGE,  # or ServiceMode.COSMOS
)
```

### 2. Using Account Name and Key

```py
store = AzureIndexStore.from_account_and_key(
    account_name="your_account_name",
    account_key="your_account_key",
    service_mode=ServiceMode.STORAGE,  # or ServiceMode.COSMOS
)
```

### 3. Using SAS Token

```py
store = AzureIndexStore.from_sas_token(
    endpoint="your_endpoint",
    sas_token="your_sas_token",
    service_mode=ServiceMode.STORAGE,  # or ServiceMode.COSMOS
)
```

### 4. Using Azure Active Directory (AAD) Token

```py
store = AzureIndexStore.from_aad_token(
    endpoint="your_endpoint",
    service_mode=ServiceMode.STORAGE,  # or ServiceMode.COSMOS
)
```

## End-to-end example:

```py
from llama_index.core import SimpleDirectoryReader, StorageContext
from llama_index.core.node_parser import SentenceSplitter
from llama_index.core import VectorStoreIndex

reader = SimpleDirectoryReader("./data/paul_graham/")
documents = reader.load_data()
nodes = SentenceSplitter().get_nodes_from_documents(documents)

storage_context = StorageContext.from_defaults(
    index_store=AzureIndexStore.from_account_and_key(
        "your_account_name",
        "your_account_key",
        service_mode=ServiceMode.STORAGE,
    ),
)

storage_context.docstore.add_documents(nodes)

keyword_table_index = SimpleKeywordTableIndex(
    nodes, storage_context=storage_context
)

vector_index = VectorStoreIndex(nodes, storage_context=storage_context)
```
llama-index-readers-huggingface-fs 0.4.1
llama-index readers huggingface fs integration
Requires Python: <4.0,>=3.9
Keywords: face, filesystem, fs, hugging, huggingface

# Hugging Face FS Loader

```bash
pip install llama-index-readers-huggingface-fs
```

This loader uses Hugging Face Hub's Filesystem API (> 0.14) to
load datasets.

Besides the existing `load_data` function, you may also choose to use
`load_dicts` and `load_df`.

## Usage

To use this loader, you need to pass in a path to a Hugging Face dataset.

```python
from pathlib import Path

from llama_index.readers.huggingface_fs import HuggingFaceFSReader

# load documents
loader = HuggingFaceFSReader()
documents = loader.load_data("datasets/dair-ai/emotion/data/data.jsonl.gz")

# load dicts
dicts = loader.load_dicts("datasets/dair-ai/emotion/data/data.jsonl.gz")

# load df
df = loader.load_df("datasets/dair-ai/emotion/data/data.jsonl.gz")
```

This loader is designed to be used as a way to load data into [LlamaIndex](https://github.com/run-llama/llama_index/).
llama-index-tools-box 0.4.1
llama-index tools box integration
Requires Python: <4.0,>=3.9

# Box Tools for Llama-Index

This repository provides a suite of Python tools for seamlessly integrating Box content with Llama Index. These tools offer an agnostic interface, allowing them to be used within any agent built within the Llama Index framework.

## Available tools include:

- **[Box Search](https://github.com/run-llama/llama_index/blob/main/llama-index-integrations/tools/llama-index-tools-box/examples/box_search.ipynb)**: Search for Box resources based on keywords and various filtering options.
- **[Search by Metadata](https://github.com/run-llama/llama_index/blob/main/llama-index-integrations/tools/llama-index-tools-box/examples/box_search_by_metadata.ipynb)**: Search for Box resources based on specific metadata fields (e.g., owner, creation date).
- **[Text Extraction](https://github.com/run-llama/llama_index/blob/main/llama-index-integrations/tools/llama-index-tools-box/examples/box_extract.ipynb)**: Extract plain text content from supported Box file formats.
- **[Box AI Prompt](https://github.com/run-llama/llama_index/blob/main/llama-index-integrations/tools/llama-index-tools-box/examples/box_ai_prompt.ipynb)**: Utilize Box AI features to analyze Box content based on user-defined prompts.
- **[Box AI Extract](https://github.com/run-llama/llama_index/blob/main/llama-index-integrations/tools/llama-index-tools-box/examples/box_ai_extract.ipynb)**: Extract structured data from Box content using pre-trained Box AI models.

By leveraging these tools, developers can empower their Llama Index agents to access, understand, and utilize valuable information stored within Box.

Check the links above for detailed usage instructions and examples for each tool.
llama-index-readers-markitdown 0.2.1
llama-index readers MarkItDown integration
Requires Python: <4.0,>=3.10

# LlamaIndex MarkItDown Reader Integration

[MarkItDown](https://github.com/microsoft/markitdown) is a powerful tool that converts various file formats to Markdown.

`llama-index-readers-markitdown` is an integration that uses MarkItDown to extract text from various file formats, supporting:

- .txt files and text-based files without extension
- .csv, .xml and .json files
- HTML files (.html)
- Presentations (.pptx)
- Word documents (.docx)
- PDF documents (.pdf)
- ZIP files (.zip)

You can install it via:

```bash
pip install llama-index-readers-markitdown
```

And you can use it in your scripts as follows:

```python
from llama_index.readers.markitdown import MarkItDownReader

reader = MarkItDownReader()
documents = reader.load_data("presentation.pptx")
```
llama-index-node-parser-dashscope 0.5.1
llama-index node_parser dashscope integration
Requires Python: <4.0,>=3.9

# LlamaIndex Node_Parser-Relational Integration: Dashscope

Transform your documents into nodes with ease using the Dashscope integration for LlamaIndex. This tool allows for precise control over chunk size, overlap size, and more, tailored for the Dashscope reader output format.

## Installation

```shell
pip install llama-index-node-parser-dashscope
```

## Quick Start

Get up and running with just a few lines of code:

```python
import json
import os
from llama_index.node_parser.relational.dashscope import (
    DashScopeJsonNodeParser,
)
from llama_index.core.ingestion import IngestionPipeline
from llama_index.core.schema import Document

# Set your Dashscope API key in the environment
os.environ["DASHSCOPE_API_KEY"] = "your_api_key_here"

documents = [
    # Prepare your documents obtained from the Dashscope reader
]

# Initialize the DashScope JsonNodeParser
node_parser = DashScopeJsonNodeParser(
    chunk_size=100, overlap_size=0, separator=" |,|，|。|？|！|\n|\?|\!"
)

# Set up the ingestion pipeline with the node parser
pipeline = IngestionPipeline(transformations=[node_parser])

# Process the documents and print the resulting nodes
nodes = pipeline.run(documents=documents, show_progress=True)
for node in nodes:
    print(node)
```

## Configuration

- API Key: You need a Dashscope API key to begin. Set it in your environment as shown in the Quick Start section.
- Document Preparation: Your documents must be in the Dashscope reader output format.
llama-index-readers-airbyte-typeform 0.4.1
llama-index readers airbyte_typeform integration
Requires Python: <4.0,>=3.9

# Airbyte Typeform Loader

```bash
pip install llama-index-readers-airbyte-typeform
```

The Airbyte Typeform Loader allows you to access different Typeform objects.

## Usage

Here's an example usage of the AirbyteTypeformReader.

```python
from llama_index.readers.airbyte_typeform import AirbyteTypeformReader

typeform_config = {
    # ...
}
reader = AirbyteTypeformReader(config=typeform_config)
documents = reader.load_data(stream_name="forms")
```

## Configuration

Check out the [Airbyte documentation page](https://docs.airbyte.com/integrations/sources/typeform/) for details about how to configure the reader.
The JSON schema the config object should adhere to can be found on Github: [https://github.com/airbytehq/airbyte/blob/master/airbyte-integrations/connectors/source-typeform/source_typeform/spec.json](https://github.com/airbytehq/airbyte/blob/master/airbyte-integrations/connectors/source-typeform/source_typeform/spec.json).

The general shape looks like this:

```python
{
    "credentials": {
        "auth_type": "Private Token",
        "access_token": "<your auth token>",
    },
    "start_date": "<date from which to start retrieving records from in ISO format, e.g. 2020-10-20T00:00:00Z>",
    "form_ids": [
        "<id of form to load records for>"
    ],  # if omitted, records from all forms will be loaded
}
```

By default all fields are stored as metadata in the documents and the text is set to the JSON representation of all the fields. Construct the text of the document by passing a `record_handler` to the reader:

```python
def handle_record(record, id):
    return Document(
        doc_id=id, text=record.data["title"], extra_info=record.data
    )


reader = AirbyteTypeformReader(
    config=typeform_config, record_handler=handle_record
)
```

## Lazy loads

The `reader.load_data` endpoint will collect all documents and return them as a list. If there are a large number of documents, this can cause issues. By using `reader.lazy_load_data` instead, an iterator is returned which can be consumed document by document without the need to keep all documents in memory.

## Incremental loads

This loader supports loading data incrementally (only returning documents that weren't loaded last time or got updated in the meantime):

```python
reader = AirbyteTypeformReader(config={...})
documents = reader.load_data(stream_name="forms")
current_state = reader.last_state  # can be pickled away or stored otherwise

updated_documents = reader.load_data(
    stream_name="forms", state=current_state
)  # only loads documents that were updated since last time
```

This loader is designed to be used as a way to load data into [LlamaIndex](https://github.com/run-llama/llama_index/).
llama-index-packs-self-discover 0.4.1
llama-index packs self_discover paper implementation
Requires Python: <4.0,>=3.9
Keywords: discover, self, self-discover, task

# Self-Discover LlamaPack

This LlamaPack implements [Self-Discover: Large Language Models Self-Compose Reasoning Structures](https://arxiv.org/abs/2402.03620) paper.

It has two stages for the given task:

1. STAGE-1:

   a. SELECT: Selects subset of reasoning Modules.

   b. ADAPT: Adapts selected reasoning modules to the task.

   c. IMPLEMENT: It gives reasoning structure for the task.

2. STAGE-2: Uses the generated reasoning structure for the task to generate an answer.

The implementation is inspired from the [codebase](https://github.com/catid/self-discover)

## CLI Usage

You can download llamapacks directly using `llamaindex-cli`, which comes installed with the `llama-index` python package:

```bash
llamaindex-cli download-llamapack SelfDiscoverPack --download-dir ./self_discover_pack
```

You can then inspect the files at `./self_discover_pack` and use them as a template for your own project!

## Code Usage

There are two ways using LlamaPack:

1. Do `download_llama_pack` to load the Self-Discover LlamaPack.
2. Directly use `SelfDiscoverPack`

### Using `download_llama_pack`

```python
from llama_index.core.llama_pack import download_llama_pack

# download and install dependencies
SelfDiscoverPack = download_llama_pack(
    "SelfDiscoverPack", "./self_discover_pack"
)

self_discover_pack = SelfDiscoverPack(verbose=True, llm=llm)
```

### Directly use `SelfRAGPack`

```python
from llama_index.packs.self_discover import SelfDiscoverPack

self_discover_pack = SelfRAGPack(llm=llm, verbose=True)
```

The run() function serves as a concise wrapper that implements the logic outlined in the "self-discover" paper, applying it to a sample task as illustrated below.

`Emma needs to prepare 50 invitations for her upcoming birthday party. She can handwrite 10 invitations in an hour. After working for 2 hours, she takes a break for 30 minutes. If she resumes writing at the same pace, how long will it take her to complete all 50 invitations?`

```python
output = pack.run("<task>")
```
llama-index-readers-linear 0.4.1
llama-index readers linear integration
Requires Python: <4.0,>=3.9
Keywords: linear

# Linear Reader

```bash
pip install llama-index-readers-linear
```

The Linear loader returns issue based on the query.

## Usage

Here's an example of how to use it

```python
from llama_index.readers.linear import LinearReader

reader = LinearReader(api_key=api_key)
query = """
    query Team {
        team(id: "9cfb482a-81e3-4154-b5b9-2c805e70a02d") {
            id
            name
            issues {
                nodes {
                    id
                    title
                    description
                    assignee {
                        id
                        name
                    }
                    createdAt
                    archivedAt
                }
            }
        }
    }
"""

documents = reader.load_data(query=query)
```

Alternately, you can also use download_loader from llama_index

```python
from llama_index.readers.linear import LinearReader

reader = LinearReader(api_key=api_key)
query = """
    query Team {
        team(id: "9cfb482a-81e3-4154-b5b9-2c805e70a02d") {
            id
            name
            issues {
                nodes {
                    id
                    title
                    description
                    assignee {
                        id
                        name
                    }
                    createdAt
                    archivedAt
                }
            }
        }
    }
"""
documents = reader.load_data(query=query)
```
llama-index-vector-stores-bigquery 0.1.1
llama-index vector_stores bigquery vector search integration
Requires Python: <4.0,>=3.9

# LlamaIndex Vector_Stores Integration: BigQuery

Vector store index using Google BigQuery.
Supports efficient storage and querying of vector embeddings using BigQuery's native vector search capabilities.
For more information, see the official [BigQuery Vector Search Documentation](https://cloud.google.com/bigquery/docs/vector-search-intro)

## 🔐 Required IAM Permissions

To use this integration, ensure your account has the following permissions:

- `roles/bigquery.dataOwner` (BigQuery Data Owner)
- `roles/bigquery.dataEditor` (BigQuery Data Editor)

## 🔧 Installation

```bash
pip install llama-index-vector-stores-bigquery
```

## 💻 Example Usage

```python
from google.cloud.bigquery import Client
from llama_index.vector_stores.bigquery import BigQueryVectorStore

client = Client()

vector_store = BigQueryVectorStore(
    table_id="my_bigquery_table",
    dataset_id="my_bigquery_dataset",
    bigquery_client=client,
)
```
llama-index-postprocessor-xinference-rerank 0.3.1
llama-index postprocessor xinference rerank integration
Requires Python: <4.0,>=3.9

# LlamaIndex Postprocessor Integration: Xinference Rerank

Xorbits Inference (Xinference) is an open-source platform to streamline the operation and integration of a wide array of AI models.

You can find a list of built-in rerank models in Xinference from its document [Rerank Models](https://inference.readthedocs.io/en/latest/models/builtin/rerank/index.html)

To learn more about Xinference in general, visit https://inference.readthedocs.io/en/stable/models/model_abilities/rerank.html

## Installation

```shell
pip install llama-index-postprocessor-xinference-rerank
```

## Usage

**Parameters Description:**

- `model`: Model uid not model name, sometimes they may be the same (e.g., `bge-reranker-base`).
- `base_url`: base url of Xinference (e.g., `http://localhost:9997`).
- `top_n`: Top n nodes to return from reranker. (default 5).

**Nodes Rerank Example**

```python
from llama_index.postprocessor.xinference_rerank import XinferenceRerank

xi_model_uid = "xinference model uid"
xi_base_url = "xinference base url"

xi_rerank = XinferenceRerank(
    top_n=5,
    model=xi_model_uid,
    base_url=xi_base_url,
)


def test_rerank_nodes(nodes, query_str):
    response = xi_rerank.postprocess_nodes(nodes, query_str)
```
llama-index-postprocessor-colbert-rerank 0.4.1
llama-index postprocessor colbert-rerank integration
Requires Python: <4.0,>=3.9

# LlamaIndex Postprocessor Integration: Colbert Rerank

[Colbert](https://github.com/stanford-futuredata/ColBERT): ColBERT is a fast and accurate retrieval model, enabling scalable BERT-based search over large text collections in tens of milliseconds.

Please `pip install llama-index-postprocessor-colbert-rerank` to install Colbert Rerank package.
llama-index-packs-node-parser-semantic-chunking 0.4.1
llama-index packs node_parser integration
Requires Python: <4.0,>=3.9
Keywords: chunk, chunking, embedding, node, parser, semantic

# Semantic Chunking Llama Pack

This LlamaPack implements the semantic chunking algorithm first proposed by Greg Kamradt in his [Five Levels of Text Splitting](https://github.com/FullStackRetrieval-com/RetrievalTutorials/blob/main/5_Levels_Of_Text_Splitting.ipynb) tutorial.

How it works:

- Split text into sentences.
- For each sentence, generate an embedding.
- Measure cosine distance between each pair of consecutive sentences.
- Get the 95% percentile cosine distance, set that as the threshold.
- Create a new chunk if the cosine distance of a sentence compared to prev. exceeds that threshold.

## CLI Usage

You can download llamapacks directly using `llamaindex-cli`, which comes installed with the `llama-index` python package:

```bash
llamaindex-cli download-llamapack LLMCompilerAgentPack --download-dir ./llm_compiler_agent_pack
```

You can then inspect the files at `./llm_compiler_agent_pack` and use them as a template for your own project!

## Code Usage

You can download the pack to a directory. **NOTE**: You must specify `skip_load=True` - the pack contains multiple files,
which makes it hard to load directly.

We will show you how to import the agent from these files!
llama-index-tools-text-to-image 0.4.1
llama-index tools text_to_image integration
Requires Python: <4.0,>=3.9

# Text to Image Tool

This tool allows Agents to use the OpenAI Image endpoint to generate and create variations of images.

## Usage

This tool has more extensive example usage documented in a Jupyter notebook [here](https://github.com/run-llama/llama_index/blob/main/llama-index-integrations/tools/llama-index-tools-text-to-image/examples/text_to_image.ipynb)

Another example showcases retrieval augmentation over a knowledge corpus with text-to-image. [Notebook](https://github.com/run-llama/llama_index/blob/main/llama-index-integrations/tools/llama-index-tools-text-to-image/examples/text_to_image-pg.ipynb).

```python
from llama_index.tools.text_to_image import TextToImageToolSpec
from llama_index.core.agent.workflow import FunctionAgent
from llama_index.llms.openai import OpenAI

openai.api_key = "sk-your-key"
tool_spec = TextToImageToolSpec()
# OR
tool_spec = TextToImageToolSpec(api_key="sk-your-key")

agent = FunctionAgent(
    tools=tool_spec.to_tool_list(),
    llm=OpenAI(model="gpt-4.1"),
)

# Context to store chat history
from llama_index.core.workflow import Context

ctx = Context(agent)


print(
    await agent.run(
        "show 2 images of a beautiful beach with a palm tree at sunset",
        ctx=ctx,
    )
)
print(await agent.run("make the second image higher quality", ctx=ctx))
```

`generate_images`: Generate images from a prompt, specifying the number of images and resolution
`show_images`: Show the images using matplot, useful for Jupyter notebooks
`generate_image_variation`: Generate a variation of an image given a URL.

This loader is designed to be used as a way to load data as a Tool in a Agent.