hyperdex 0.1.1
GitHub Repository Catalog: Fetching, indexing, and organizing READMEs.
License: MIT
Requires Python: >=3.10

# hyperdex

> [!CAUTION]
> This project is currently for personal use only. A public-facing version with improved documentation, examples, and user-friendly interfaces is planned for future development. See the [Package Transformation Plan](docs/plans/package-transformation-plan.md) for details on how this will evolve.
>
> For now, Just don't use it!

[![Update GitHub Forks](https://github.com/shaneholloman/hyperdex/actions/workflows/update_github_forks.yml/badge.svg)](https://github.com/shaneholloman/hyperdex/actions/workflows/update_github_forks.yml) [![Update GitHub Sources](https://github.com/shaneholloman/hyperdex/actions/workflows/update_github_sources.yml/badge.svg)](https://github.com/shaneholloman/hyperdex/actions/workflows/update_github_sources.yml)

Your GitHub Repository Catalog: A comprehensive tool for fetching, indexing, and organizing READMEs from GitHub repositories.

- [`cello-repos`](./github-project-readmes-shane/github-project-readmes-cello/README.md)
- [`shane-repos`](./github-project-readmes-shane/README.md)

## Scripts

This repository contains several Python scripts for managing repository information:

**README Fetching & Indexing:**

- `gh_repo_fetch_index_shane.py`: Fetches READMEs from your personal GitHub repositories (public and private, excluding forks) and generates an index file (`github-project-readmes-shane/README.md`) with repository metadata and statistics.
- `gh_repo_fetch_forks_shane.py`: Fetches READMEs from your personal *forked* GitHub repositories and generates an index file (`github-project-forks-shane/README.md`) with repository metadata and statistics.
- `gh_repo_fetch_index_cello.py`: Fetches READMEs from the `CelloCommunications` organization repositories authored by you (based on the first commit) and generates a similar index file (`github-project-readmes-cello/README.md`).

**Release Management (for personal repos):**

- `gh_repo_release_latest_shane.py`: Reads `github-project-readmes-shane/repositories.csv`, finds the latest semantic version tag for each personal repository, and creates a GitHub Release with auto-generated notes if one doesn't already exist for that tag.
- `gh_repo_release_initial_shane.py`: Reads `github-project-readmes-shane/repositories.csv` and creates an initial `v0.1.0` tag and release for any personal repository that currently has no tags.
- `gh_repo_release_latest_cello.py`: Fetches repositories authored by you in the `CelloCommunications` org, finds the latest semantic version tag for each, and creates a GitHub Release with auto-generated notes if one doesn't already exist. Includes a `--dry-run` option.
- `gh_repo_release_initial_cello.py`: Fetches repositories authored by you in the `CelloCommunications` org and creates an initial `v0.1.0` tag and release for any that currently have no tags. Includes a `--dry-run` option.
- `gh_repo_update_readmes.py`: Reads repository names from a CSV file (e.g., `github-project-readmes-shane/repositories.csv`), finds corresponding local README files, and updates the remote READMEs using the GitHub API.
- `gh_repo_setup_secret.py`: Sets up a GitHub Personal Access Token as a repository secret (e.g., `GH_PAT` for workflows).

## Features

- Fetches READMEs from GitHub repositories
- Filters by authorship (first commit author)
- Generates a Markdown index with links to all fetched READMEs
- Includes creation and update dates for each repository

## Performance Optimization

A significant optimization has been implemented using GitHub's GraphQL API to reduce API calls and improve efficiency. See [optimization.md](docs/optimization.md) for details on:

- The original vs. optimized approach
- Technical implementation with GraphQL
- Performance benefits
- Implementation considerations

## Development Approach

### Terminal Testing First

**IMPORTANT:** Before implementing any scripting solution, ALWAYS test your approach directly in the terminal first. This principle was critical to discovering the GraphQL optimization in this project.

For example, before implementing the GraphQL solution in Python:

1. Basic GraphQL query was tested directly with GitHub CLI:

   ```sh
   gh api graphql -f query='query { organization(login: "CelloCommunications") { ... } }'
   ```

2. Once the query worked, it was refined interactively:

   ```sh
   gh api graphql -f query='...' | jq '.data.organization.repositories.nodes[] | select(...)'
   ```

3. Only after confirming the approach worked in the terminal was it implemented in Python.

This terminal-first approach allows you to:

- Verify API responses without writing complex code
- Iterate quickly on query structure
- Identify potential issues early
- Understand exactly what data you're working with

### Installation

You can install hyperdex directly from PyPI:

```sh
# Using UV (the only supported method)
uv tool install hyperdex
```

> **IMPORTANT**: This project strictly follows a UV-native workflow. The use of pip is strictly prohibited. See [UV Workflow](docs/uv-workflow.md) for details.

### Usage

#### As a Command-Line Tool

After installation, you can use the command-line tools directly:

```sh
# Fetch/Index personal READMEs (excluding forks)
hyperdex-fetch-shane

# Fetch/Index personal FORKED READMEs
hyperdex-fetch-forks-shane

# Fetch/Index organization READMEs authored by you
hyperdex-fetch-cello

# Create releases for latest tags on personal repos (if missing)
hyperdex-release-latest-shane

# Create initial v0.1.0 release for untagged personal repos
hyperdex-release-initial-shane

# Create latest releases for organization repos (Dry Run)
hyperdex-release-latest-cello --dry-run

# Create initial v0.1.0 release for untagged organization repos (Dry Run)
hyperdex-release-initial-cello --dry-run

# Update README files across multiple repositories using a CSV list
hyperdex-update-readmes -c path/to/your/repositories.csv -d path/to/local/readmes/

# Setup GitHub PAT as a repository secret
hyperdex-setup-secret
```

#### From Source

Clone this repository:

```sh
git clone https://github.com/shaneholloman/hyperdex.git
cd hyperdex
```

Run the desired script from the `tools/` directory:

```sh
# Fetch/Index personal READMEs (excluding forks)
python3 tools/gh_repo_fetch_index_shane.py

# Fetch/Index personal FORKED READMEs
python3 tools/gh_repo_fetch_forks_shane.py

# Fetch/Index Cello READMEs authored by you
python3 tools/gh_repo_fetch_index_cello.py

# Create releases for latest tags on personal repos (if missing)
python3 tools/gh_repo_release_latest_shane.py

# Create initial v0.1.0 release for untagged personal repos
python3 tools/gh_repo_release_initial_shane.py

# Create latest releases for Cello repos (Dry Run)
python3 tools/gh_repo_release_latest_cello.py --dry-run

# Create initial v0.1.0 release for untagged Cello repos (Dry Run)
python3 tools/gh_repo_release_initial_cello.py --dry-run

# Update README files across multiple repositories using a CSV list
python3 tools/gh_repo_update_readmes.py -c path/to/your/repositories.csv -d path/to/local/readmes/

# Setup GitHub PAT as a repository secret
python3 tools/gh_repo_setup_secret.py
```

The fetcher scripts will:

1. Fetch repositories from GitHub
2. Filter by authorship (for the Cello script)
3. Download READMEs from each repository
4. Generate an index file with links to all fetched READMEs

## Requirements

- Python 3.10+ (required for `|` type hint syntax)
- GitHub CLI (gh) installed and authenticated
- `packaging` library for Python (install with `uv install packaging`) for release scripts

## Output

The scripts will create subdirectories with fetched READMEs and an index file:

- `github-project-readmes-shane/`: For personal repositories (excluding forks)
- `github-project-forks-shane/`: For personal forked repositories
- `github-project-readmes-cello/`: For organization repositories

Each directory contains:

- Individual README files renamed according to repository
- An `index.md` file with links to all READMEs and metadata

## License

MIT