repro-audit 0.1.1
A static analysis tool for auditing ML paper reproducibility
License: MIT
Requires Python: >=3.8

# repro-audit

A deterministic static analysis tool for auditing ML papers and code repositories for reproducibility.

## Scope

`repro-audit` parses a paper (PDF) and a code repository to check for compliance with reproducibility standards. It generates a report and a badge based on purely static analysis.

**Core Checks:**
- **Seeds**: Are seeds mentioned in the paper and explicitly set in the code?
- **Hyperparameters**: Are hyperparameters mentioned in the paper configurable in the code?
- **Data**: Is data handling explicit and split logic clear?
- **Dependencies**: Are dependencies pinned?


## Installation

```bash
python -m venv .env
source .env/bin/activate
pip install -e .
```

If you prefer, install from PyPI once released:

```bash
pip install repro-audit
```

## Usage

```bash
repro-audit paper.pdf --repo ./my-repo
```

Command flags such as `--baseline-paper` and `--baseline-repo` enable diff mode to compare the current audit against an earlier release of the paper and repository. Reports and badges are emitted as `repro_report.*` and `badge.svg`.

## Outputs

- `repro_report.md`: Human-readable report.
- `repro_report.json`: Machine-readable results.
- `badge.svg`: Status badge.

## Advanced Coverage

- **Hyperparameter traceability:** matches every paper hyperparameter mention to argparse/config locations so reviewers can verify where defaults are controlled.
- **Dataset provenance:** tracks dataset names/URLs in the paper, links them to scripts/configs, and surfaces transforms/licenses so data sources are transparent.
- **Runtime environment capture:** parses Dockerfiles, docker-compose, CI workflows, and CUDA-related dependencies to certify the claimed execution stack.
- **Responsible research sections:** ensures the paper or README exposes citation, ethics/fairness, or limitations sections to document responsible-use considerations.
- **Diff mode:** compare the audit against a historical baseline to highlight regressions in reproducibility.

## Diff Mode Example

```bash
repro-audit paper.pdf --repo ./my-repo --baseline-paper paper-v0.pdf --baseline-repo ../releases/v0.1
```

Generates `repro_report_diff.md` with rule-level status changes and the score delta.

## Example run

An example paper and audit output live in `examples/`. To rerun the sample audit:

```bash
repro-audit examples/sample_paper.tex --repo tests/synthetic_repo
```

The command produces `repro_report.*` and `badge.svg` in the repo root again (already shown inside `examples/` to illustrate expected output).

## License

`repro-audit` is distributed under the [MIT License](./LICENSE). See `LICENSE` for the full text and attribution requirements.

## Contributing & Troubleshooting

- **Contributing**: Fork the repo, make changes in a feature branch, and submit a pull request describing the improvements plus any new tests or verification steps.
- **Troubleshooting**: Re-run the sample audit (`repro-audit examples/sample_paper.tex --repo tests/synthetic_repo`) to regenerate `repro_report.*` and `badge.svg`; compare against your last run with `repro_report_diff.md` to spot regressions. If a rule behaves unexpectedly, inspect the JSON/Markdown results before filing an issue so you can describe the audit context and relevant files.

## Publishing to PyPI

- Bump `version` in `setup.py`, push a git tag/release, and ensure the repository secret `PYPI_API_TOKEN` is set; the GitHub workflow `.github/workflows/publish-pypi.yml` publishes on release.
- For manual publishing: `python -m build` then `twine upload dist/*` (use a virtual environment to avoid system Python restrictions).