amzn-bbam-dev 1.1.0
Bayesian Bias Assessment Model for RCT vs Observational Model Comparisons
Requires Python: >=3.12
Keywords: adjustment, bayesian, bias, observational, pymc, rct

# BBAM - Bayesian Bias Assessment Model

Amazon Confidential 

Collaboration: [mtirsk@](https://phonetool.amazon.com/users/mtirsk) (GME), [jdmoral@](https://phonetool.amazon.com/users/jdmoral) (MAPS), [skrainka@](https://phonetool.amazon.com/users/skrainka) (MAPS), [khawandc@](https://phonetool.amazon.com/users/khawandc) (CBA), [janekb@](https://phonetool.amazon.com/users/janekb) (GME Science), [shabhus@](https://phonetool.amazon.com/users/shabhus) (GMAC Econ Engineering)

BBAM is a Python package that implements Hierarchical Bayesian Models (HBMs) to estimate observational model bias by comparing observational models against Randomized Controlled Trials (RCTs). It provides a unified implementation for evaluating and calibrating observational models across Amazon teams.

This package implements the recommended meta-analysis model for CCM reporting described [here](https://quip-amazon.com/t4FlA8AAqh26/RCT-Meta-Analysis-for-CCM-Example-of-Deliverable-Updated-103024).

The science papers: [Standard HBM](https://drive.corp.amazon.com/documents/mtirsk@/public/RCT_Meta_Analysis.pdf) | [Scale-Aware HBM](https://drive.corp.amazon.com/documents/jdmoral@/papers/Beyond%20One-Size-Fits-All%20A%20Scale-Aware%20Bayesian%20Framework%20for%20Accurate%20Marketing%20Causality.pdf)

## Quick Start

### Get Running in 3 Steps

1. **Set up Brazil workspace**:
   ```bash
   brazil ws create --name BBAM --versionset MAPSScienceLibraries/development
   cd BBAM && brazil ws use --package BBAM
   ```

2. **Build and activate**:
   ```bash
   brazil-build
   source .venv/bin/activate
   ```

3. **Run example**:
   ```bash
   python examples/scripts/run_bbam_minimal.py
   ```

### Basic Usage

```python
import pandas as pd
from bbam import BbamAnalyzer

# Load your data
data = pd.read_csv("rct_observational_pairs.csv")

# Fit model and get results
model = BbamAnalyzer(model_type="standard")
model.fit(data)
results = model.get_results()

# Calibrate new observational data
df_unpaired = pd.read_csv("unpaired_observational_data.csv")
calibrated_results = model.calibrate(df_unpaired)
```

## Model Types

### Standard Model (`model_type="standard"`)
Estimates bias in observational models using RCT ground truth.

**Required columns**: `obs_model_est`, `obs_model_est_se`, `experiment_est`, `experiment_est_se`

### Scale-Aware Model (`model_type="scaled"`)
Estimates bias while accounting for experiment scale metrics (impressions, spend, subjects).

**Required columns**: Standard columns plus `experiment_scale`, `experiment_scale_se`

**Key concept**: **Treatment Effect Intensity (TEI)** = Treatment Effect ÷ Experiment Scale
- Enables comparison across experiments of different scales
- Example: Campaign with 1M impressions showing 0.02 effect has same TEI as 10M impression campaign with 0.2 effect

### RCT-Only Model (`model_type="rct_only"`)
Meta-analysis of RCT data for pooled effect estimates.

**Required columns**: `experiment_est`, `experiment_est_se`

## Installation Options

### Option 1: Production Use (Recommended)
Add to your project's `pyproject.toml`:
```toml
[project]
dependencies = ["amzn-bbam>=1.1.0"]
```

### Option 2: Development Setup
```bash
# Clone and build locally
brazil ws create --name BBAM --versionset MAPSScienceLibraries/development
cd BBAM && brazil ws use --package BBAM
brazil-build
```

### Option 3: Direct Installation
```bash
# Authenticate to CodeArtifact
aws codeartifact login --tool pip --domain amazon --repository shared
pip install amzn-bbam
```
(For initial setup, in case you haven't, see __Option D: Direct pip Install from Amazon CodeArtifact__ in [CHEATSHEET.md](https://code.amazon.com/packages/BBAM/blobs/mainline/--/CHEATSHEET.md#))


## Key Features

- **Unified Interface**: Single `BbamAnalyzer` for all workflows
- **Flexible Configuration**: YAML-based with sensible defaults
- **Robust Statistics**: Hierarchical Bayesian modeling with PyMC
- **Built-in Calibration**: Apply bias corrections to new data
- **Comprehensive Testing**: 388 tests with 84% coverage
- **Legacy Support**: Backward compatibility maintained

## Configuration

BBAM uses YAML configuration with these main sections:

```yaml
data_driven_priors: true          # Automatic prior setting
sampling_settings:
  draws: 2000
  chains: 4
model_settings:
  bias_spec: "BOTH_INT_MULT"      # Bias structure
  likelihood_distribution: "normal"
```

See `examples/configs/` for complete examples.

## Calibration

Both Standard and Scale-Aware models support calibration for new observational data:

```python
# Fit model on paired data
model.fit(paired_data)

# Apply bias corrections to unpaired data
calibrated_results = model.calibrate(df_unpaired)

# Access results
calibrated_values = calibrated_results.values
calibrated_summary = calibrated_results.summary
results_df = calibrated_results.to_dataframe()
```

## Examples & Documentation

- **Notebooks**: `examples/notebooks/bbam_minimal.ipynb`
- **Scripts**: `examples/scripts/run_bbam_minimal.py`
- **Sample Data**: `examples/data/`
- **Detailed Setup**: [CHEATSHEET.md](CHEATSHEET.md)
- **Development**: [DEVELOPMENT.md](DEVELOPMENT.md)
- **Architecture**: `doc/design/`

## Testing

```bash
# Run all tests
brazil-build test

# Run specific test categories
brazil-build test --addopts="test/unit/"        # Fast unit tests
brazil-build test --addopts="test/integration/" # Full workflows
brazil-build test --addopts="test/specialized/" # Advanced scenarios
```
Also, once in a built environment, you can test with hatch and pyenv: 
```bash
# After building the code (or modifying the test scripts)
source .hatch/bin/activate 
hatch run python -m pytest  -v --tb=short
```


## Package Information

- **Published Name**: `amzn-bbam` (production), `amzn-bbam-dev` (development)
- **Repository**: AWS CodeArtifact (`amazon/shared`)
- **Pipeline**: [MAPSScienceLibraries](https://pipelines.amazon.com/pipelines/MAPSScienceLibraries)
- **Discovery**: [Peruse](https://prod.peruse.builder-tools.aws.dev/shared/search/pypi/amzn-bbam)
- **Version**: 1.1.0
- **Python**: 3.12+

## Support

- **Documentation**: `doc/` directory for detailed specifications
- **Examples**: Review notebooks and scripts in `examples/`
- **Issues**: Report through your team's standard channels

Internal Amazon package - see your team's guidelines for usage and distribution policies.