nomy-data-models 0.35.9
Data model definitions for Nomy wallet analysis data processing
Requires Python: <4.0,>=3.9

# Nomy Data Models

This repository contains the data model definitions for Nomy wallet analysis data processing. These models are shared across multiple services:

- nomy-data-service
- nomy-data-processor
- nomy-data-ingestor

## Overview

The Nomy Data Models repository serves as a single source of truth for all data structures used in the Nomy wallet analysis ecosystem. By centralizing these models, we ensure consistency across different services and reduce duplication of code.

## Features

- **Cross-language support**: Models can be used in both Python and Rust
- **Versioned releases**: Proper versioning to manage dependencies
- **Consistent data structures**: Ensures data integrity across the entire pipeline
- **Centralized schema management**: Single source of truth for all data models
- **SQLAlchemy ORM models**: Database-ready models with full ORM support
- **Automatic Rust code generation**: Python SQLAlchemy models are automatically converted to Rust structs

## Installation

### Python

```bash
# Install from PyPI
pip install nomy-data-models

# Install from source
pip install -e .
```

### Rust

Add to your `Cargo.toml`:

```toml
[dependencies]
nomy-data-models = { version = "0.1.0", git = "https://github.com/bcnmy/nomy-data-models" }
```

## Usage

### Python

```python
from nomy_data_models.models import WalletState, ServiceState, RawTrade, EnrichedTrade, Position

# Use the models with SQLAlchemy
from sqlalchemy import create_engine
from sqlalchemy.orm import Session

engine = create_engine("sqlite:///nomy.db")
with Session(engine) as session:
    wallet_state = WalletState(
        wallet_address="0x1234567890abcdef1234567890abcdef12345678",
        chain_id=1
    )
    session.add(wallet_state)
    session.commit()
```

### Rust

```rust
use nomy_data_models::models::{WalletState, ServiceState, RawTrade, EnrichedTrade, Position};

// Use the model
let wallet_state = WalletState {
    wallet_address: "0x1234567890abcdef1234567890abcdef12345678".to_string(),
    chain_id: 1,
    // ... other fields
};
```

## Development Setup

### Pre-commit Hooks

This repository uses pre-commit hooks to ensure code quality and consistency. After cloning the repository, install the pre-commit hooks:

```bash
# Make the script executable
chmod +x ./scripts/install-hooks.sh

# Install the hooks
./scripts/install-hooks.sh
```

The pre-commit hooks will automatically:

- Format code with Black
- Sort imports with isort
- Run Python tests
- Generate and verify Rust models
- Format Rust code with rustfmt
- Run Rust linting with Clippy

**Important**: Always ensure the pre-commit hooks are installed and up-to-date. They are essential for maintaining code quality and preventing CI failures.

## Development

### Prerequisites

- Python 3.8+
- Rust 1.50+
- Poetry (for Python dependency management)
- Cargo (for Rust dependency management)

### Setup

1. Clone the repository
2. Install Python dependencies: `poetry install`
3. Build Rust package: `cargo build`
4. Install git hooks: `./scripts/install-hooks.sh`

### Git Hooks

This repository includes git hooks to automate certain tasks:

- **pre-commit**: Automatically generates Rust models from Python models and verifies they build correctly before each commit

To install the git hooks, run:

```bash
./scripts/install-hooks.sh
```

This ensures that all Rust models are kept in sync with their Python counterparts and that they build correctly.

### Adding New Models

1. Add the model definition to the appropriate Python file in `nomy_data_models/models/`
2. Update the `__init__.py` files to export the new model
3. Run the code generation script: `python scripts/generate_rust.py`
4. Verify the generated Rust code in `src/models/`
5. Add tests for both Python and Rust implementations

### Testing

```bash
# Run Python tests
pytest tests/python

# Run Rust tests
cargo test
```

## Contributing

Please follow these steps when contributing:

1. Create a new branch for your feature
2. Add or modify models as needed
3. Ensure tests pass for both Python and Rust
4. Submit a pull request

## Package Deployment

### Preconditions

Before deploying a new version, ensure you have:

1. Updated the version number in `Cargo.toml` for the Rust package
2. Updated the version number in `pyproject.toml` for the Python package

### Automated Deployment

When your changes are merged into the main branch, the CI pipeline will automatically:

1. Deploy the Python package to PyPI (https://pypi.org/)
2. Deploy the Rust package to crates.io (https://crates.io/)

The deployment process is handled automatically by the CI pipeline, so you don't need to perform any manual steps after merging to main.

## Python-to-Rust Conversion

This repository includes a utility for automatically converting Python SQLAlchemy models to Rust structs. This ensures that the data models are consistent across both languages.

### How it works

1. Python SQLAlchemy models are defined in the `nomy_data_models/models` directory
2. The `nomy_data_models/py_to_rust.py` module provides functions for converting these models to Rust
3. The `scripts/generate_rust.py` script can be used to generate Rust code from the command line

### Usage

To generate Rust models from Python SQLAlchemy models:

```bash
python scripts/generate_rust.py [output_dir]
```

If `output_dir` is not specified, the default is `src/models`.

### Testing

To test the conversion process:

```bash
python scripts/test_conversion.py
```

This will generate Rust code in the `scripts/test_output/models` directory and run various tests to ensure the conversion is working correctly.