vector-dataloader 1.2.7
Production-grade embedding loader for CSV data to vector stores with multiple providers.
License: MIT
Requires Python: <3.14,>=3.11

vector-dataloader
vector-dataloader is a robust and extensible Python library for loading CSV data from local files or AWS S3 into vector stores (Postgres, FAISS, Chroma) with embedding generation. It supports multiple embedding providers (AWS Bedrock, Google Gemini, Sentence-Transformers, OpenAI) and offers flexible embedding modes for scalable data processing. Visit the GitHub repository for source code and examples.
🚀 Features
Core Data Loading

Data Sources: Load data from local CSV files or AWS S3.
Embedding Modes: Supports combined or separated embedding generation.
Embedding Providers: AWS Bedrock, Google Gemini, Sentence-Transformers, OpenAI.
Vector Stores: Postgres (with pgvector), FAISS (in-memory), Chroma (persistent).
Data Updates: Handles new, updated, or removed rows with soft delete support.
Scalability: Batch processing, retries, and connection pooling for efficient operations.
Extensibility: Modular plugin-style architecture for providers and stores.
Validation: Enforces schema, type, and null checks for data integrity.

DataMove Use Case (New!)

Production-grade data migration from CSV to PostgreSQL without embedding generation.
Flexible schema validation with strict and flexible modes.
Automatic table creation and schema evolution.
Case-sensitivity conflict detection and prevention.
Comprehensive error handling with automatic rollback.
S3 integration with automatic loader selection.
Performance optimization with configurable batch processing.
Dry-run validation for previewing operations.

📦 Installation
Prerequisites

Python: Version 3.8 or higher.
Visual Studio Build Tools: Required for C++ dependencies (e.g., FAISS). Download from Visual Studio Build Tools and ensure the "Desktop development with C++" workload is installed.
pip or uv: Package manager for installing dependencies.

Step-by-Step Installation

Install the Core PackageInstall the minimal package without optional dependencies:
pip install vector-dataloader
# or
uv add vector-dataloader


Install Optional DependenciesInstall only the dependencies for the providers and stores you need:



Combination
Command
Notes



ChromaDB
pip install vector-dataloader[chroma]
Required for ChromaVectorStore.


FAISS
pip install vector-dataloader[faiss]
Required for FaissVectorStore.


Google Gemini
pip install vector-dataloader[gemini]
Required for GeminiEmbeddingProvider.


Sentence-Transformers
pip install vector-dataloader[sentence-transformers]
Required for SentenceTransformersProvider.


OpenAI
pip install vector-dataloader[openai]
Required for OpenAIProvider.


AWS Bedrock
pip install vector-dataloader[bedrock]
Required for BedrockEmbeddingProvider.


All Features
pip install vector-dataloader[all]
Installs all optional dependencies.


Example: To use Chroma with Gemini:
pip install vector-dataloader[chroma,gemini]


Verify InstallationConfirm the package is installed:
pip show vector-dataloader



⚙️ Usage
For detailed usage examples, refer to the examples folder in the GitHub repository: https://github.com/RatelClaw/dataload.  

Loading and Data Validation: See examples/test_all_scenarios.py for comprehensive scenarios covering data loading and validation.  
Embedding Features: Explore main_chroma_st.py, main_chroma_gemini.py, and other main files for specific embedding provider implementations.

All examples use asynchronous execution for efficiency. Ensure the input CSV file (e.g., data_to_load/sample.csv or data_to_load/sample_2.csv) exists with appropriate columns (e.g., id, name, description or Index, Name, Description).
🛠️ Configuration
Environment Variables
Configure the library using a .env file in your project root or system environment variables.Example .env:
# Google Gemini API Key
GOOGLE_API_KEY=your_google_api_key_here

# Postgres Configuration
LOCAL_POSTGRES_HOST=localhost
LOCAL_POSTGRES_PORT=5432
LOCAL_POSTGRES_DB=your_db_name
LOCAL_POSTGRES_USER=postgres
LOCAL_POSTGRES_PASSWORD=your_password

# AWS Configuration (for Bedrock/S3)
AWS_REGION=ap-southeast-1
SECRET_NAME=your_secret_name

Notes:

For AWS Bedrock or S3, configure AWS credentials using:aws configure
aws configure set aws_secret_access_key


Set use_aws=True in DBConnection for AWS integration.
Ensure the input CSV file matches the expected schema.

DataMove Use Case
The DataMove use case provides production-grade data migration from CSV files to PostgreSQL databases without embedding generation, ideal for ETL pipelines and data migration scenarios.
Quick Start
Refer to the examples/data_move_comprehensive_example.py for a production-ready example.
Key Features

Automatic Table Creation: Creates tables from CSV schema.
Schema Validation: Strict (existing_schema) and flexible (new_schema) modes.
S3 Integration: Automatic detection of S3 URIs.
Error Handling: Comprehensive validation with rollback on failures.
Performance: Configurable batch processing and memory management.
Dry Run: Preview operations without making changes.

Documentation

Quick Start Guide: Get started in 5 minutes.
API Documentation: Complete API reference.
Troubleshooting Guide: Common issues and solutions.
Comprehensive Examples: Production-ready examples.

📚 License
MIT LicenseCopyright (c) 2025 Shashwat Roy  
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.