gazoo-research-utils 2.1.1
Utilities for the Analysis of Gazoo Research Data
Requires Python: >=3.11

# GazooResearchUtils
GazooResearchUtils is a Python library designed for analyzing medical data from GazooResearch. It provides tools for filtering, aggregating, and visualizing longitudinal patient data with support for time-based analyses anchored to specific events.

## Installation

```bash
pip install gazoo-research-utils
```

## Data Structure

Gazoo Research data follows a denormalized structure with this hierarchy:
- **id**: Patient identification
- **collection**: A collection of related tags (e.g., 'c61' for oncology data, '0' for demographics)
- **tag**: Name of the tag (e.g., 'dob', 'psa', 'surgery')
- **tag_id**: Unique identifier for the tag
- **field**: Field name (e.g., 'date', 'value', 'units', 'mrn')
- **data_type**: Data type ('text', 'categorical', 'number', 'date')
- **value**: The actual value
- **phi**: 0 or 1, indicating if the data element is protected health information

### Example Data Structure

```csv
id,collection,tag,tag_id,field,data_type,value,phi
111111,0,id,#6bed55ed-f844-4c88-ab74-af0e74aa2547,mrn,text,111111,1
111111,0,last-name,bd7e898c-9732-4320-8227-a1b90cc8c80e,value,text,Bing,1
111111,0,dob,#5578f001-b34b-4465-b699-8c8ff6c11d89,date,date,1952-12-17,0
111111,c61,surgery,#6082ea75-06c3-4082-9c69-918d0df8c619,date,date,2010-01-01,0
111111,c61,surgery,#6082ea75-06c3-4082-9c69-918d0df8c619,type,categorical,prostatectomy,0
111111,c61,psa,64987467-bcc2-4c23-a943-899467d93981,date,date,2019-02-22,0
111111,c61,psa,64987467-bcc2-4c23-a943-899467d93981,value,number,0.28,0
111111,c61,psa,64987467-bcc2-4c23-a943-899467d93981,units,categorical,ng/ml,0
```

## Core Concepts

### Filters

Filters define criteria for selecting patient data. A filter is specified as a dictionary with these fields:

```python
filter = {
    'collection': str,    # Optional: collection name
    'tag': str,           # Optional: tag name
    'field': str,         # Optional: field name
    'exact': [str, ...],  # Optional: exact value matches
    'between': [float, float]  # Optional: numeric range
}
```

### Anchors

Anchors define reference points for time-based analysis. They identify specific events (like surgery or a lab test) from which other measurements are compared.

#### Anchor Format

An anchor follows the same structure as a filter:

```python
anchor = {
    'collection': str,    # Optional
    'tag': str,           # Required: tag name
    'field': str,         # Optional: field name
    'exact': [str, ...],  # Optional
    'between': [float, float]  # Optional
}
```

For time-based analyses, anchors should include a date field (e.g., `field='date'` or `field='start-date'`).

### Tag Sequences

Multiple anchors can be specified as a sequence. All anchors in the sequence must occur within the specified `time_delta`:

```python
anchor_sequence = [
    {'tag': 'surgery'},
    {'tag': 'radiation'}
]
```

### Instance Selection

When multiple instances of an anchor sequence exist for a patient, use `instance` to specify which occurrence to use:

- `instance = 0` - First occurrence of the sequence
- `instance = 1` - Second occurrence
- `instance = -1` - Last occurrence

## Getting Started

### Loading Data

```python
from gazoo_research_utils import load_data, validate_gazoo_research_df

# Load data from CSV
df = load_data("path/to/prostate-data.csv")

# Validate the DataFrame structure
validate_gazoo_research_df(df)
```

### Filtering Data

#### Simple Filters

```python
from gazoo_research_utils import get_tags_where_filter, filter

# Filter by collection
collection_filter = {'collection': 'c61'}
data_c61 = get_tags_where_filter(df, collection_filter)

# Filter by tag
psa_filter = {'tag': 'psa'}
psa_data = get_tags_where_filter(df, psa_filter)

# Filter by field with exact match
psa_value_filter = {'tag': 'psa', 'field': 'value'}
psa_values = get_tags_where_filter(df, psa_value_filter)

# Filter numeric values within range
psa_range_filter = {'tag': 'psa', 'field': 'value', 'between': [0.2, 5.0]}
psa_in_range = get_tags_where_filter(df, psa_range_filter)

# Multiple filters (AND logic)
multi_filters = [
    {'tag': 'gleason-grade-group', 'field': 'value', 'exact': [4, 5]},
    {'tag': 'psa', 'field': 'value', 'between': [0.2, 0.5]}
]
matched_patients = filter(df, multi_filters)
```

#### Using Filter Objects

```python
from gazoo_research_utils.models.filter_model import Filter

# Create a Filter object
filter_obj = Filter(collection='c61', tag='psa', field='value', between=[0.2, 5.0])
result = get_tags_where_filter(df, filter_obj)
```

## Anchor-Based Analysis

### Getting Anchors

```python
from gazoo_research_utils import get_anchors
from datetime import timedelta

# Single anchor
anchor = [{'tag': 'surgery', 'field': 'date'}]
anchors_df = get_anchors(df, anchor, time_delta=0, instance=0)

# Anchor sequence (surgery followed by radiation within 60 days)
anchor_sequence = [
    {'tag': 'surgery', 'field': 'date'},
    {'tag': 'radiation', 'field': 'date'}
]
anchors_df = get_anchors(
    df, 
    anchor_sequence, 
    time_delta=60,  # 60 days max between anchors
    instance=0      # First occurrence
)

# Using timedelta for more precision
anchors_df = get_anchors(
    df, 
    [{'tag': 'surgery'}], 
    time_delta=timedelta(days=5, hours=3),  # 5 days and 3 hours
    instance=0
)
```

### Filtering Data Around Anchors

```python
from gazoo_research_utils import filter_around_anchor

# Get patients with PSA value within 30 days before to 7 days after surgery
anchors_df = get_anchors(df, [{'tag': 'surgery'}], time_delta=0, instance=0)
filters = [{'collection': 'c61', 'tag': 'psa', 'field': 'value'}]
result = filter_around_anchor(df, anchors_df, filters, [-30.0, 7.0])

# Get patients with pT category within 1 day of surgery
filters = [
    {'collection': 'c61', 'tag': 'pT', 'field': 'T', 'exact': ['2a', '2b']}
]
result = filter_around_anchor(df, anchors_df, filters, [-1.0, 1.0])
```

### Time Delta

The `time_delta` parameter sets the maximum number of **days** or **timedelta** allowed between sequential anchors. This is only relevant when specifying multiple anchors.

```python
anchor_sequence = [
    {'tag': 'surgery'}, 
    {'tag': 'radiation'}
]

# Using float (days)
anchors_df = get_anchors(df, anchor_sequence, time_delta=60.0, instance=0)

# Using timedelta
from datetime import timedelta
anchors_df = get_anchors(df, anchor_sequence, time_delta=timedelta(days=60), instance=0)
anchors_df = get_anchors(df, anchor_sequence, time_delta=timedelta(days=5, hours=3), instance=0)
```

This is useful for filtering out scenarios like salvage radiotherapy, which typically occurs months or years later rather than immediate adjuvant treatment.

## Time Series Analysis

### Getting Tag Time Series

```python
from gazoo_research_utils import get_tag_time_series

# Get time series data for PSA
df_plot = get_tag_time_series(df, tag='psa')

# With anchor normalization
anchors_df = get_anchors(df, [{'tag': 'surgery'}], time_delta=90.0, instance=0)
df_plot = get_tag_time_series(df, tag='psa', anchors=anchors_df)
```

The returned DataFrame includes:
- `id`: Patient identifier
- `date` or `start-date`: The measurement date
- `value`: The measurement value
- Additional columns (e.g., `units`, `mrn`)
- `time-delta`: Days from the first measurement (or anchor date if provided)

### Visualization

```python
from gazoo_research_utils import line_plot, categorical_plot

# Line plot for numerical data
fig = line_plot(
    df_plot, 
    field='value', 
    x_scale='year',  # or 'day'
    xlabel='Years Since First Measurement', 
    ylabel='PSA (ng/mL)'
)

# Categorical plot
fig = categorical_plot(
    df_plot,
    field='T',
    y_scale=['0', '1', '2a', '2b', '3a'],
    x_scale='day',
    xlabel='Days Since First Measurement',
    ylabel='T Category'
)
```

## Patient-Level Transformations

### Pivot Data

```python
from gazoo_research_utils import pivot

# Get PSA data
psa_df = get_tags_where_filter(df, {'tag': 'psa'})

# Pivot to human-readable format
psa_pivoted = pivot(psa_df)

# Access PSA values for all patients
psa_values = psa_pivoted["c61:psa:value"]
```

### Calculate Age

```python
from gazoo_research_utils import calculate_age

# Calculate age at anchor date
anchors_df = get_anchors(df, [{'tag': 'surgery'}], time_delta=0, instance=0)
df_with_age = calculate_age(df, anchors_df)
```

### Multi-lesion Transformation

```python
from gazoo_research_utils import multilesion_transforation

# Transform patient-level data to multi-target level
df_multi = multilesion_transforation(df)
```

## Statistical Analysis

### Describe Fields Around Anchors

```python
from gazoo_research_utils import describe_fields, plot_table1

# Get anchors
anchors_df = get_anchors(df, [{'tag': 'surgery'}], time_delta=90.0, instance=0)

# Extract field values within 90 days before to 1 day after surgery
fields = [
    {'collection': 'c61', 'tag': 'psa', 'field': 'value'},
    {'tag': 'pT', 'field': 'T'}
]
results = describe_fields(df, anchors_df, fields, [-90.0, 1.0])

# Plot results
fig = plot_table1([results])
```

### Kaplan-Meier Survival Analysis

```python
from gazoo_research_utils import kaplan_meier, plot_km_curves

# Get anchors
anchors_df = get_anchors(df, [{'tag': 'surgery'}], time_delta=0, instance=0)

# Define event tags
event_tags = [
    {'tag': 'progression', 'field': 'date'}
]

# Calculate Kaplan-Meier data (time in months)
km = kaplan_meier(df, anchors_df, event_tags, time_unit="months")

# Plot survival curve
fig = plot_km_curves(km, time_column="time", event_column="event")
```

### Time in Months

```python
from gazoo_research_utils import time_in_months
from datetime import datetime

start_date = datetime.strptime("2019-01-01", "%Y-%m-%d")
end_date = datetime.strptime("2020-06-15", "%Y-%m-%d")
months = time_in_months(start_date, end_date)  # Returns 17
```

## Utility Functions

### Data Dictionary

```python
from gazoo_research_utils import get_data_dictionary

# Get data dictionary
dictionary = get_data_dictionary(df)
```

### Get Parameters

```python
from gazoo_research_utils import get_parameters

# Get parameter options
parameters = get_parameters(df)
```

### Patient and Tag Iterators

```python
from gazoo_research_utils import patient_iterator, tag_iterator

# Apply function to each patient
def patient_fn(patient_df):
    # Process patient data
    return patient_df

result = patient_iterator(df, patient_fn)

# Apply function to each tag
def tag_fn(tag_df):
    # Process tag data
    return tag_df

result = tag_iterator(df, tag_fn)
```

### Start at Anchor

```python
from gazoo_research_utils import start_at_anchor

# Remove all data before anchor date
anchors_df = get_anchors(df, [{'tag': 'surgery'}], time_delta=0, instance=0)
df_after_anchor = start_at_anchor(df, anchors_df)
```

### Link by Field Value

```python
from gazoo_research_utils import link_by_field_value

# Get all tags with the same field value as a reference tag
tag_link = {'collection': 'c61', 'tag': 'target', 'field': 'mrn'}
linked_data = link_by_field_value(df, tag_link, instance=0)
```

## Complete Example

```python
from gazoo_research_utils import (
    load_data, 
    get_anchors, 
    filter_around_anchor, 
    get_tag_time_series,
    line_plot,
    describe_fields,
    plot_table1,
    kaplan_meier,
    plot_km_curves
)
from datetime import timedelta

# Load data
df = load_data("prostate-data.csv")

# 1. Get patients who had surgery followed by radiation within 90 days
anchor_sequence = [
    {'tag': 'surgery', 'field': 'date'},
    {'tag': 'radiation', 'field': 'date'}
]
anchors_df = get_anchors(df, anchor_sequence, time_delta=90, instance=0)

# 2. Get PSA values 30 days before to 7 days after surgery
filters = [{'collection': 'c61', 'tag': 'psa', 'field': 'value'}]
psa_around_surgery = filter_around_anchor(df, anchors_df, filters, [-30.0, 7.0])

# 3. Get time series for PSA normalized to surgery date
df_time_series = get_tag_time_series(df, tag='psa', anchors=anchors_df)

# 4. Visualize PSA time series
fig = line_plot(
    df_time_series,
    field='value',
    x_scale='year',
    xlabel='Years Since Surgery',
    ylabel='PSA (ng/mL)'
)

# 5. Describe patient characteristics at surgery
fields = [
    {'tag': 'age', 'field': 'value'},
    {'tag': 'pT', 'field': 'T'},
    {'tag': 'gleason-grade-group', 'field': 'value'}
]
results = describe_fields(df, anchors_df, fields, [-1.0, 1.0])
fig_table = plot_table1([results])

# 6. Kaplan-Meier survival analysis
event_tags = [{'tag': 'progression', 'field': 'date'}]
km = kaplan_meier(df, anchors_df, event_tags, time_unit="months")
fig_km = plot_km_curves(km)
```

## API Reference

### Data Loading & Validation

| Function | Description |
|----------|-------------|
| `load_data(file_path)` | Load data from CSV with type conversion |
| `validate_gazoo_research_df(df)` | Validate DataFrame structure |

### Filtering

| Function | Description |
|----------|-------------|
| `get_tags_where_filter(df, filter)` | Get rows matching filter criteria |
| `filter(df, filters)` | Filter patients by multiple criteria |

### Anchor Functions

| Function | Description |
|----------|-------------|
| `get_anchors(df, anchors, time_delta, instance)` | Get anchor dates for patients |
| `filter_around_anchor(df, anchors_df, filters, search_range)` | Filter data around anchor dates |
| `start_at_anchor(df, anchors)` | Remove data before anchor dates |
| `calculate_age(df, anchors)` | Calculate patient age at anchor |

### Time Series

| Function | Description |
|----------|-------------|
| `get_tag_time_series(df, tag, anchors)` | Get time series for a tag |
| `pivot(df)` | Pivot data to wide format |
| `line_plot(df, field, x_scale, xlabel, ylabel)` | Line plot for numerical data |
| `categorical_plot(df, field, y_scale, ...)` | Plot for categorical data |

### Statistical Analysis

| Function | Description |
|----------|-------------|
| `describe_fields(df, anchors, fields, search_range)` | Extract field values around anchors |
| `plot_table1(df_describes)` | Create Table 1 visualization |
| `kaplan_meier(df, anchors, event_tags, time_unit)` | Kaplan-Meier survival analysis |
| `plot_km_curves(km, time_column, event_column)` | Plot Kaplan-Meier curves |
| `time_in_months(start, end)` | Calculate months between dates |

### Utility Functions

| Function | Description |
|----------|-------------|
| `get_data_dictionary(df)` | Get data dictionary |
| `get_parameters(df)` | Get parameter options |
| `patient_iterator(df, patient_fn)` | Apply function to each patient |
| `tag_iterator(df, tag_fn)` | Apply function to each tag |
| `link_by_field_value(df, tag_link, instance)` | Link tags by field value |
| `multilesion_transforation(df)` | Transform to multi-target analysis |

## Related

For more information, see the [Gazoo Research](https://gazooresearch.com) documentation.