dcmbench 0.1.2
A comprehensive benchmarking framework for discrete choice models
License: MIT
Requires Python: >=3.8
Keywords: discrete choice, benchmarking, transportation, econometrics, machine learning

# Discrete Choice Model Benchmarking (DCMBench)

A comprehensive Python package for benchmarking, analyzing, and validating discrete choice models for transportation mode choice analysis. DCMBench provides a unified framework for comparing different model specifications, conducting sensitivity analysis, and visualizing results.

## Installation

You can install DCMBench using pip:

```bash
pip install dcmbench
```

## Key Features

### Model Estimation and Benchmarking

- **Multiple Model Types**: Support for Multinomial Logit (MNL), Nested Logit (NL), and Mixed Logit (ML) models
- **Standardized Metrics**: Compare models using log-likelihood, rho-squared, prediction accuracy, and market share
- **Cross-validation**: Evaluate model performance on training and testing datasets
- **Visualization**: Generate comparative plots showing model performance across different metrics

```python
from dcmbench.model_benchmarker import Benchmarker
from dcmbench.datasets import fetch_data

# Load dataset (automatically downloads if not in local cache)
data = fetch_data("swissmetro_dataset")

# Define models and run benchmark
benchmarker = Benchmarker()
benchmarker.register_model(models, "Model Name")
results = benchmarker.run_benchmark(data, choice_column="CHOICE")
benchmarker.print_comparison()
```

### Advanced Analysis Capabilities

- **Sensitivity Analysis**: Evaluate how model predictions change with variations in key variables
  - Simulate changes in travel times, costs, and other attributes
  - Generate plots showing the evolution of market shares under different scenarios

- **Individual-Level Parameters**: For Mixed Logit models, calculate and visualize:
  - Individual-specific parameter distributions using Bayesian approaches
  - Value of Time (VOT) distributions across the population
  - Heterogeneity in preference structures

- **Model Calibration**: Automatically calibrate Alternative Specific Constants (ASCs) to match observed market shares

### Visualization and Reporting

- **Market Share Analysis**: Compare predicted vs. observed mode shares
- **Performance Plots**: Visualize how different models perform across multiple datasets
- **Parameter Distributions**: Plot distributions of random parameters and derived metrics like VOT
- **Sensitivity Curves**: Show how predicted mode shares change with variations in key variables

## Supported Datasets

- **Swissmetro** (`swissmetro_dataset`): Swiss inter-city travel mode choice
- **London Transport** (`ltds_dataset`): London Travel Demand Survey with urban mode choices
- **ModeCanada** (`modecanada_dataset`): Canadian inter-city travel dataset

Datasets are automatically downloaded from the [dcmbench-datasets](https://github.com/carlosguirado/dcmbench-datasets) repository on first use and cached locally:

```python
from dcmbench.datasets import fetch_data

# Use default cache location (~/.dcmbench/datasets)
data = fetch_data("swissmetro_dataset")

# Specify custom cache location
data = fetch_data("swissmetro_dataset", local_cache_dir="/path/to/cache")

# Get features and target separately
X, y = fetch_data("swissmetro_dataset", return_X_y=True)
```

## Example Applications

### Benchmarking Multiple Models

The package includes tools to benchmark multiple model types across different datasets:

```python
# Run benchmark_all_models.py to compare models across datasets
python benchmark_all_models.py
```

This generates comparative visualizations showing how different model types perform across datasets, plotting metrics like choice accuracy and market share accuracy against model fit.

### Sensitivity Analysis

Analyze how changes in key variables affect predicted mode shares:

```python
# Run sensitivity analysis on ModeCanada models
python sensitivity_analysis.py
```

This creates plots showing the evolution of mode shares as you modify variables like:
- Travel costs for different modes
- Travel times
- Service frequencies

### Individual Parameter Analysis

For Mixed Logit models, analyze individual-level parameters and VOT:

```python
# Generate individual parameter distributions for ModeCanada
python plot_individual_parameters_canada.py
```

This calculates individual-specific parameters using Bayesian conditioning and produces:
- Distributions of time and cost parameters
- Value of Time (VOT) distributions
- Summary statistics for preference heterogeneity

## Requirements

- Python >=3.8
- NumPy >=2.0.0
- Pandas >=2.0.0
- Biogeme >=3.2.14
- Matplotlib >=3.0.0
- Requests >=2.25.0
- SciPy (for statistical functions)
- Seaborn (for advanced visualizations)

## License

This project is licensed under the MIT License - see the LICENSE file for details.

## Contributing

We welcome contributions! Please see our contributing guidelines for details.

### Adding New Datasets

To add a new dataset to the DCMBench ecosystem:

1. Fork the [dcmbench-datasets](https://github.com/carlosguirado/dcmbench-datasets) repository
2. Add your dataset following the structure guidelines in the repository's CONTRIBUTING.md file
3. Submit a pull request to the dcmbench-datasets repository
4. Update the metadata.json file in the main DCMBench package to include your dataset information

This design allows the package to remain lightweight while providing access to a growing collection of transportation mode choice datasets.