semvec/pypi-packages huggingface dataset:
llama-index-readers-youtube-transcript 0.4.1
llama-index readers youtube transcript integration
Requires Python: <4.0,>=3.9
Keywords: video

# Youtube Transcript Loader

```bash
pip install llama-hub-youtube-transcript

pip install llama-index-readers-youtube-transcript
```

This loader fetches the text transcript of Youtube videos using the `youtube_transcript_api` Python package.

## Usage

To use this loader, you will need to first `pip install youtube_transcript_api`.

Then, simply pass an array of YouTube links into `load_data`:

```python
from llama_index.readers.youtube_transcript import YoutubeTranscriptReader

loader = YoutubeTranscriptReader()
documents = loader.load_data(
    ytlinks=["https://www.youtube.com/watch?v=i3OYlaoj-BM"]
)
```

Supported URL formats: + youtube.com/watch?v={video_id} (with or without 'www.') + youtube.com/embed?v={video_id} (with or without 'www.') + youtu.be/{video_id} (never includes www subdomain)

To programmatically check if a URL is supported:

```python
from llama_index.readers.youtube_transcript.utils import is_youtube_video

is_youtube_video("https://youtube.com/watch?v=j83jrh2")  # => True
is_youtube_video("https://vimeo.com/272134160")  # => False
```

This loader is designed to be used as a way to load data into [LlamaIndex](https://github.com/run-llama/llama_index/).
llama-index-readers-nougat-ocr 0.4.1
llama-index readers nougat_ocr integration
Requires Python: <4.0,>=3.9
Keywords: academic papers, ocr, pdf

# Nougat OCR loader

```bash
pip install llama-index-readers-nougat-ocr
```

This loader reads the equations, symbols, and tables included in the PDF.

Users can input the path of the academic PDF document `file` which they want to parse. This OCR understands LaTeX math and tables.

## Usage

Here's an example usage of the PDFNougatOCR.

```python
from llama_index.readers.nougat_ocr import PDFNougatOCR

reader = PDFNougatOCR()

pdf_path = Path("/path/to/pdf")

documents = reader.load_data(pdf_path)
```

## Miscellaneous

An `output` folder will be created with the same name as the pdf and `.mmd` extension.
llama-index-vector-stores-moorcheh 0.2.1
llama-index vector_stores moorcheh integration
Requires Python: <4.0,>=3.10

# LlamaIndex Vector_Stores Integration: Moorcheh

Welcome to the Moorcheh Vector Store that integrates Llama-Index.

This module introduces support for [**Moorcheh**](https://github.com/mjfekri/moorcheh-python-sdk), a semantic vector database developed by **EdgeAI Innovations**. Moorcheh enables fast and intelligent document retrieval using hybrid scoring and generative answering capabilities.
The integration is implemented in accordance with the standard vector store interface defined by **LlamaIndex** and supports all core methods including `add`, `query`, `delete`, and `generate_answer`.

To see the integration in action, refer to the demonstration notebook: [Google Colab Demo](https://colab.research.google.com/drive/1iUoMpNYcJxmu1xTySMNJZBPbOQIkUEEs?usp=sharing).

## Getting started

To begin using the Moorcheh vector store, make sure to install the necessary packages:

```
pip install llama_index
pip install moorcheh_sdk
```

## Example Usage

Here is a simple example demonstrating how to use the Moorcheh integration with LlamaIndex:

```
from llama_index.core import VectorStoreIndex
from llama_index.llama_index_integrations.vector_stores.llama_index_vector_stores_moorcheh.llama_index.vector_stores-moorcheh import base, init, utils

api_key = os.environ["MOORCHEH_API_KEY"]

documents = SimpleDirectoryReader("./your-directory").load_data()
__all__ = ["MoorchehVectorStore"]

# Creates a Moorcheh Vector Store with the following parameters
# For text-based namespaces, set namespace_type to "text" and vector_dimension to None
# For vector-based namespaces, set namespace_type to "vector" and vector_dimension to the dimension of your uploaded vectors
vector_store = MoorchehVectorStore(api_key=api_key, namespace="llamaindex_moorcheh", namespace_type="text", vector_dimension=None, add_sparse_vector=False, batch_size=100)

storage_context = StorageContext.from_defaults(vector_store=vector_store)
index = VectorStoreIndex.from_documents(
    documents, storage_context=storage_context
)

query_engine = index.as_query_engine()
response = query_engine.query("Which company has had the highest revenue in 2025 and why?")

display(Markdown(f"<b>{response}</b>"))
print("\n\n================================\n\n", response, "\n\n================================\n\n")
```
llama-index-packs-amazon-product-extraction 0.4.1
llama-index packs amazon_product_extraction integration
Requires Python: <4.0,>=3.9
Keywords: amazon, extraction, product

# Amazon Product Extraction Pack

This LlamaPack provides an example of our Amazon product extraction pack.

It loads in a website URL, screenshots the page. Then we use OpenAI GPT-4V + prompt engineering to extract the screenshot into a structured JSON output.

Check out the [notebook here](https://github.com/run-llama/llama-hub/blob/main/llama_hub/llama_packs/amazon_product_extraction/product_extraction.ipynb).

## CLI Usage

You can download llamapacks directly using `llamaindex-cli`, which comes installed with the `llama-index` python package:

```bash
llamaindex-cli download-llamapack AmazonProductExtractionPack --download-dir ./amazon_product_extraction_pack
```

You can then inspect the files at `./amazon_product_extraction_pack` and use them as a template for your own project.

## Code Usage

You can download the pack to a the `./amazon_product_extraction_pack` directory:

```python
from llama_index.core.llama_pack import download_llama_pack

# download and install dependencies
AmazonProductExtractionPack = download_llama_pack(
    "AmazonProductExtractionPack", "./amazon_product_extraction_pack"
)
```

From here, you can use the pack, or inspect and modify the pack in `./amazon_product_extraction_pack`.

Then, you can set up the pack like so:

```python
# create the pack
# get documents from any data loader
amazon_product_extraction_pack = SentenceWindowRetrieverPack(
    amazon_product_page,
)
```

The `run()` function is a light wrapper around `program()`.

```python
response = amazon_product_extraction_pack.run()
display(response.dict())
```

You can also use modules individually.

```python
# get pydantic program
program = amazon_product_extraction_pack.openai_program

# get multi-modal LLM
mm_llm = amazon_product_extraction_pack.openai_mm_llm
```
llama-index-retrievers-duckdb-retriever 0.5.1
llama-index retrievers duckdb-retriever integration
Requires Python: <4.0,>=3.9

# LlamaIndex Retrievers Integration: DuckDBRetriever

`pip install llama-index-retrievers-duckdb-retriever.`

## What is this?

This is a BM25 Retriever for DuckDB that can be used with LlamaIndex to enable full-text search.

## How to use

This was created with reference to [DuckDB Full-Text Search Extension](https://duckdb.org/docs/extensions/full_text_search), so it's mostly the same.

Please refer to that.

However, while `DuckDBVectorStore` is an VectorStore, `DuckDBRetriever` is a Retriever.

DuckDBRetriever Example:

```python
from llama_index.retrievers.duckdb_retriever import DuckDBRetriever

retriever = DuckDBRetriever(database_name="vector.db", persist_dir="duckdb")
nodes = retriever.retrieve("retrieve_query")
```
llama-index-packs-ragatouille-retriever 0.5.1
llama-index packs ragatouille_retriever integration
Requires Python: <4.0,>=3.9
Keywords: rag, ragatouille, retriever

# RAGatouille Retriever Pack

RAGatouille is a [cool library](https://github.com/bclavie/RAGatouille) that lets you use e.g. ColBERT and other SOTA retrieval models in your RAG pipeline. You can use it to either run inference on ColBERT, or use it to train/fine-tune models.

This LlamaPack shows you an easy way to bundle RAGatouille into your RAG pipeline. We use RAGatouille to index a corpus of documents (by default using colbertv2.0), and then we combine it with LlamaIndex query modules to synthesize an answer with an LLM.

A full notebook guide can be found [here](https://github.com/run-llama/llama-hub/blob/main/llama_hub/llama_packs/ragatouille_retriever/ragatouille_retriever.ipynb).

## CLI Usage

You can download llamapacks directly using `llamaindex-cli`, which comes installed with the `llama-index` python package:

```bash
llamaindex-cli download-llamapack RAGatouilleRetrieverPack --download-dir ./ragatouille_pack
```

You can then inspect the files at `./` and use them as a template for your own project!

## Code Usage

You can download the pack to a `./ragatouille_pack` directory:

```python
from llama_index.core.llama_pack import download_llama_pack

# download and install dependencies
RAGatouilleRetrieverPack = download_llama_pack(
    "RAGatouilleRetrieverPack", "./ragatouille_pack"
)
```

From here, you can use the pack, or inspect and modify the pack in `./ragatouille_pack`.

Then, you can set up the pack like so:

```python
# create the pack
ragatouille_pack = RAGatouilleRetrieverPack(
    docs,  # List[Document]
    llm=OpenAI(model="gpt-3.5-turbo"),
    index_name="my_index",
    top_k=5,
)
```

The `run()` function is a light wrapper around `query_engine.query`.

```python
response = ragatouille_pack.run("How does ColBERTv2 compare to BERT")
```

You can also use modules individually.

```python
from llama_index.core.response.notebook_utils import display_source_node

retriever = ragatouille_pack.get_modules()["retriever"]
nodes = retriever.retrieve("How does ColBERTv2 compare with BERT?")

for node in nodes:
    display_source_node(node)

# try out the RAG module directly
RAG = ragatouille_pack.get_modules()["RAG"]
results = RAG.search(
    "How does ColBERTv2 compare with BERT?", index_name=index_name, k=4
)
results
```
llama-index-embeddings-modelscope 0.5.1
llama-index embeddings modelscope integration
Requires Python: <3.12,>=3.9

# LlamaIndex Embedding Integration: ModelScope

## Installation

To install the required package, run:

```bash
!pip install llama-index-embeddings-modelscope
```

## Basic Usage

### Initialize the ModelScopeLLM

To use the ModelScopeEmbedding model, create an instance by specifying the model name and revision:

```python
from llama_index.embeddings.modelscope.base import ModelScopeEmbedding

model = ModelScopeEmbedding(
    model_name="iic/nlp_gte_sentence-embedding_chinese-base",
    model_revision="master",
)
```

### Generate Embedding

To generate a text embedding for a query, use the `get_query_embedding` method or `get_text_embedding` method:

```python
rsp = model.get_query_embedding("Hello, who are you?")
print(rsp)

rsp = model.get_text_embedding("Hello, who are you?")
print(rsp)
```

### Generate Batch Embedding

To generate a text embedding for a batch of text, use the `get_text_embedding_batch` method:

```python
rsp = model.get_text_embedding_batch(
    ["Hello, who are you?", "I am a student."]
)
print(rsp)
```
llama-index-readers-dashscope 0.4.1
llama-index readers dashscope integration
Requires Python: <4.0,>=3.9

# LlamaIndex Readers Integration: Dashscope

## Installation

```shelll
pip install llama-index-readers-dashscope
```

## Usage

```python
from llama_index.readers.dashscope.base import DashScopeParse
from llama_index.readers.dashscope.utils import ResultType

file_list = [
    # your files (accept doc, docx, pdf)
]

parse = DashScopeParse(result_type=ResultType.DASHCOPE_DOCMIND)
documents = parse.load_data(file_path=file_list)
```

## Reader Setting:

A full list of retriever settings/kwargs is below:

- api_key: Optional[str] -- Your dashscope API key, which can be passed in through environment variables or parameters.
  The parameter settings will override the results from the environment variables
- workspace_id: Optional[str] -- Your dashscope workspace_id, which can be passed in through environment variables or
  parameters. The parameter settings will override the results from the environment variables
- base_url: Optional[str] -- The base url for the Dashscope API. The default value is "https://dashscope.aliyuncs.com".
  The parameter settings will override the results from the environment variables.
- result_type: Optional[ResultType] -- The result type for the parser. The default value is ResultType.DASHCOPE_DOCMIND.
- num_workers: Optional[int] -- The number of workers to use sending API requests for parsing. The default value is 4,
  greater than 0, less than 10.
- check_interval: Optional[int] -- The interval in seconds to check if the parsing is done. The default value is 5.
- max_timeout: Optional[int] -- The maximum timeout in seconds to wait for the parsing to finish. The default value is 3600.
- verbose: Optional[bool] -- Whether to print the progress of the parsing. The default value is True.
- show_progress: Optional[bool] -- Show progress when parsing multiple files. The default value is True.
- ignore_errors: Optional[bool] -- Whether or not to ignore and skip errors raised during parsing. The default value is
  True.

## Reader Input:

- file_path: Union[str, List[str]] -- The file path or list of file paths to parse.

## Reader Output:

- List[llama_index.core.schema.Document] -- The list of documents parsed from the file.
  - text: str -- The text of the document from DASHCOPE_DOCMIND.
llama-index-readers-arango-db 0.4.1
llama-index readers arango db integration
Requires Python: <4.0,>=3.9

# LlamaIndex Readers Integration: ArangoDB

```bash
pip install llama-index-readers-arango-db
```

This loader loads documents from [ArangoDB](https://github.com/arangodb/arangodb?tab=readme-ov-file#arangodb). The user specifies an ArangoDB instance to
initialize the reader. They then specify the collection name and query parameters to
fetch the relevant docs.

## Usage

Here's an example usage of the SimpleArangoDBReader.

```python
import os

from llama_index.readers.arango_db import SimpleArangoDBReader

host = "<host>"
db_name = "<db_name>"
collection_name = "<collection_name>"
# query_dict is passed into db.collection.find()
query_dict = {}
# Attribute of interests to load, by default ["text"]
field_names = ["title", "description"]
reader = SimpleArangoDBReader(host)  # or pass ArangoClient
documents = reader.load_data(
    username,
    password,
    db_name,
    collection_name,
    query_dict=query_dict,
    field_names=field_names,
)
```

A demo notebook is available [here](https://colab.research.google.com/github/arangodb/interactive_tutorials/blob/master/notebooks/example_output/Langchain_Full_output.ipynb).

This loader is designed to be used as a way to load data into [LlamaIndex](https://github.com/run-llama/llama_index/).
llama-index-tools-cassandra 0.4.1
llama-index tools Apache Cassandra® integration
Requires Python: <4.0,>=3.9

# Cassandra Database Tools

## Overview

The Cassandra Database Tools project is designed to help AI engineers efficiently integrate Large Language Models (LLMs) with Apache Cassandra® data. It facilitates optimized and safe interactions with Cassandra databases, supporting various deployments like Apache Cassandra®, DataStax Enterprise™, and DataStax Astra™.

## Key Features

- **Fast Data Access:** Optimized queries ensure most operations complete in milliseconds.
- **Schema Introspection:** Enhances the reasoning capabilities of LLMs by providing detailed schema information.
- **Compatibility:** Supports various Cassandra deployments, ensuring wide applicability.
- **Safety Measures:** Limits operations to SELECT queries and schema introspection to prioritize data integrity.

## Installation

Ensure your system has Python installed and proceed with the following installations via pip:

```bash
pip install python-dotenv cassio llama-index-tools-cassandra
```

Create a `.env` file for environmental variables related to Cassandra and Astra configurations, following the example structure provided in the notebook.

## Environment Setup

- For Cassandra: Configure `CASSANDRA_CONTACT_POINTS`, `CASSANDRA_USERNAME`, `CASSANDRA_PASSWORD`, and `CASSANDRA_KEYSPACE`.
- For DataStax Astra: Set `ASTRA_DB_APPLICATION_TOKEN`, `ASTRA_DB_DATABASE_ID`, and `ASTRA_DB_KEYSPACE`.

## How It Works

The toolkit leverages the Cassandra Query Language (CQL) and integrates with LLMs to provide an efficient query path determination for the user's requests, ensuring best practices for querying are followed. Using functions, the LLMs decision making can invoke the tool instead of designing custom queries. The result is faster and efficient access to Cassandra data for agents.

## Tools Included

- **`cassandra_db_schema`**: Fetches schema information, essential for the agent’s operation.
- **`cassandra_db_select_table_data`**: Allows selection of data from a specific keyspace and table.
- **`cassandra_db_query`**: An experimental tool that accepts fully formed query strings from the agent.

## Example Usage

Initialize the CassandraDatabase and set up the agent with the tools provided. Query the database by interacting with the agent as shown in the example [notebook](https://docs.llamaindex.ai/en/stable/examples/tools/cassandra/).
llama-index-llms-cleanlab 0.6.1
llama-index llms cleanlab integration
Requires Python: <4.0,>=3.9

# LlamaIndex Llms Integration: Cleanlab

## Overview

Integrate with Cleanlab's Trustworthy Language Model (TLM) APIs.

## Installation

```bash
pip install llama-index-llms-cleanlab
```

## Example

With environmental variables.

```.env
CLEANLAB_API_KEY=your_api_key
```

```python
from llama_index.llms.cleanlab import CleanlabTLM

# Initialize Cleanlab's TLM without explicitly passing the API key and base
llm = CleanlabTLM()

# Make a query to the LLM
response = llm.complete("Explain the importance of open source LLMs")

print(response)
```

Without environmental variables

```python
from llama_index.llms.cleanlab import CleanlabTLM

# Set up the CleanlabTLM's class with the required API key and quality preset
llm = CleanlabTLM(
    quality_preset="best",  # supported quality presets are: 'best','high','medium','low','base'
    api_key="your_api_key",
)

# Call the complete method with a query
response = llm.complete("Explain the importance of open source LLMs")

print(response)
```
llama-index-readers-zulip 0.4.1
llama-index readers zulip integration
Requires Python: <4.0,>=3.9

## Zulip Loader

The Zulip Loader is a Python script that allows you to load data from Zulip streams using a Zulip bot's API token. It fetches messages from specified streams or all streams if none are specified, and returns a list of documents with the stream content.

### Prerequisites

Create a Zulip bot and obtain its API token. Follow the instructions in the Zulip documentation to create a bot and get the API key (token).

Set the ZULIP_TOKEN environment variable to your Zulip bot's API token:

```bash
export ZULIP_TOKEN="your-zulip-bot-api-token"
```

### Installation

You can install the Zulip Reader via pip:

```bash
pip install llama-index-readers-zulip
```

### Usage

Use the ZulipReader class to load data from Zulip streams:

```python
from zulip_loader import ZulipReader

# Initialize the ZulipReader with the bot's email and Zulip domain
reader = ZulipReader(
    zulip_email="your-bot-email@your-zulip-domain.zulipchat.com",
    zulip_domain="your-zulip-domain.zulipchat.com",
)

# Load data from all streams
data = reader.load_data(reader.get_all_streams())

# Load data from specific streams
stream_names = ["stream1", "stream2"]
data = reader.load_data(stream_names)
# This will return a list of documents containing the content of the specified streams.
```

For more customization, you can pass the `reverse_chronological` parameter to the load_data() method to indicate the order of messages in the output.

This loader is designed to be used as a way to load data into
[LlamaIndex](https://github.com/run-llama/llama_index/tree/main/llama_index) and/or subsequently
used as a Tool in a [LangChain](https://github.com/hwchase17/langchain) Agent.
llama-index-packs-chroma-autoretrieval 0.4.1
llama-index packs chroma_autoretrieval integration
Requires Python: <4.0,>=3.10
Keywords: chroma, retrieval, vector

# Chroma AutoRetrieval Pack

This LlamaPack inserts your data into chroma and instantiates an auto-retriever, which will use the LLM at runtime to set metadata filtering, top-k, and query string.

## CLI Usage

You can download llamapacks directly using `llamaindex-cli`, which comes installed with the `llama-index` python package:

```bash
llamaindex-cli download-llamapack ChromaAutoretrievalPack --download-dir ./chroma_pack
```

You can then inspect the files at `./chroma_pack` and use them as a template for your own project!

## Code Usage

You can download the pack to a the `./chroma_pack` directory:

```python
from llama_index.core.llama_pack import download_llama_pack

# download and install dependencies
ChromaAutoretrievalPack = download_llama_pack(
    "ChromaAutoretrievalPack", "./chroma_pack"
)
```

From here, you can use the pack, or inspect and modify the pack in `./chroma_pack`.

Then, you can set up the pack like so:

```python
# setup pack arguments
from llama_index.core.vector_stores import MetadataInfo, VectorStoreInfo

vector_store_info = VectorStoreInfo(
    content_info="brief biography of celebrities",
    metadata_info=[
        MetadataInfo(
            name="category",
            type="str",
            description=(
                "Category of the celebrity, one of [Sports Entertainment, Business, Music]"
            ),
        ),
    ],
)

import chromadb

client = chromadb.EphemeralClient()

nodes = [...]

# create the pack
chroma_pack = ChromaAutoretrievalPack(
    collection_name="test",
    vector_store_info=vector_store_index,
    nodes=nodes,
    client=client,
)
```

The `run()` function is a light wrapper around `query_engine.query()`.

```python
response = chroma_pack.run("Tell me a bout a Music celebritiy.")
```

You can also use modules individually.

```python
# use the retriever
retriever = chroma_pack.retriever
nodes = retriever.retrieve("query_str")

# use the query engine
query_engine = chroma_pack.query_engine
response = query_engine.query("query_str")
```
llama-index-llms-maritalk 0.5.1
llama-index llms maritalk integration
Requires Python: <3.12,>=3.9

# LlamaIndex Llms Integration: Maritalk

MariTalk is an assistant developed by the Brazilian company [Maritaca AI](https://www.maritaca.ai). MariTalk is based on language models that have been specially trained to understand Portuguese well.

## Installation

First, install the Llama Index library (and all its dependencies) using the following command:

```
$pip install llama-index llama-index-llms-maritalk
```

## API Key

You will need and API key that can be obtained from chat.maritaca.ai (\"Chaves da API\" section).

## Examples

Examples of usage are presented at [llamahub.ai](https://docs.llamaindex.ai/en/stable/examples/llm/maritalk/).
llama-index-embeddings-yandexgpt 0.4.1
llama-index embeddings yandexgpt integration
Requires Python: <4.0,>=3.9

# LlamaIndex Embeddings Integration: YandexGPT

YandexGPT Embeddings provides text embedding service using the Yandex Cloud API. It supports both synchronous and asynchronous methods for embedding texts and documents(queries), making it flexible and suitable for various applications in natural language processing (NLP) and machine learning.

To learn more about Yandex Cloud API and embedding principles, visit https://yandex.cloud/en/docs/foundation-models/concepts/embeddings

## Installation

```bash
pip install llama-index-embeddings-yandexgpt
```

## Usage

```python
from llama_index.embeddings.yandexgpt import YandexGPTEmbedding
```

**Initialization Parameters:**

- `api_key`: The API key for Yandex Cloud. This key is required for authenticating requests.
- `folder_id`: The folder ID for Yandex Cloud.

```python
embeddings = YandexGPTEmbedding(
    api_key="your-api-key",
    folder_id="your-folder-id",
)
```

## Example

See the [example notebook](../../../docs/docs/examples/embeddings/yandexgpt.ipynb) for a detailed walkthrough of using YandexGPT embeddings with LlamaIndex.