cove-tools 1.0.0
Holland Casino AWS multi-account analysis tools
License: MIT
Requires Python: >=3.10

# cove-tools

This repository contains tools for analyzing AWS resources across multiple accounts.

## Usage

### Preferred Method: Using Make

The recommended way to run any tool in this repository is through the Makefile:

```bash
# Generate all reports
make reports

# Generate a specific report
make route53_health_report
```

This method automatically handles environment setup and ensures all dependencies are available.

### Direct Script Execution

While using make is preferred, you can run scripts directly if needed. However, this requires additional setup:

1. First time setup:
   ```bash
   ./Setup.sh
   ```

2. Set up your shell environment (required each time):
   ```bash
   # For bash
   eval "$(direnv export bash)"
   
   # For zsh
   eval "$(direnv export zsh)"
   ```

3. Then you can run the scripts:
   ```bash
   ./bin/cove_route53_health [options]
   ```

Note: Direct script execution is more complex and error-prone compared to using make. Use make unless you have a specific reason not to.

## Tools

### cove_lambda_runtimes

This tool analyzes Lambda runtimes across AWS accounts and generates a report of their usage. It can produce both console output and HTML reports.

#### Features

- Analyze Lambda runtimes across multiple AWS accounts
- Filter results by programming language or specific runtime version
- Generate full reports or focus on deprecated/end-of-support runtimes
- Output results as console text or HTML report

#### Usage

The recommended way to run this tool is through make:

```bash
make lambda_runtimes_report
```

For more details on options and examples, see the [cove_lambda_runtimes section](#cove_lambda_runtimes-1) below.

If you need to run the script directly (not recommended), ensure you've followed the [Direct Script Execution](#direct-script-execution) setup steps first.

### cove_codebuild_runtimes

This tool analyzes CodeBuild runtimes across AWS accounts and generates a report of their usage. It produces text reports with different levels of detail.

#### Features

- Analyze CodeBuild runtimes across multiple AWS accounts
- Generate reports with different levels of detail
- Cache data for faster subsequent runs

#### Usage

The recommended way to run this tool is through make:

```bash
make codebuild_runtimes_report
```

If you need to run the script directly (not recommended), ensure you've followed the [Direct Script Execution](#direct-script-execution) setup steps first.

#### Options

These options apply only when running the script directly:
- `--use-cache`: Use cached data instead of fetching new data

Note: The make target automatically handles caching for optimal performance.

#### Examples

1. Generate a report of CodeBuild runtimes (recommended):
   ```bash
   make codebuild_runtimes_report
   ```

#### Output

The script generates four text reports:

1. `codebuild_runtimes_report1.txt`: Overview of all CodeBuild runtimes
2. `codebuild_runtimes_report2.txt`: Overview of CodeBuild runtimes per AWS account
3. `codebuild_runtimes_report3.txt`: Detailed overview of CodeBuild runtimes per AWS account
4. `codebuild_runtimes_report4.txt`: Overview of outdated CodeBuild runtimes per AWS account

### cove_ses_identities

This tool collects and reports on all SES identities across multiple AWS accounts.

#### Features

- Collect SES identities (both email addresses and domains) across multiple AWS accounts
- Generate a comprehensive report of all SES identities
- Cache data for faster subsequent runs

#### Usage

The recommended way to run this tool is through make:

```bash
make ses_identities_report
```

If you need to run the script directly (not recommended), ensure you've followed the [Direct Script Execution](#direct-script-execution) setup steps first.

### cove_ses_users

This tool analyzes SES user activity across AWS accounts by correlating CloudTrail events and SES logs.

#### Features

- Track SES user activity across all accounts
- Analyze CloudTrail events for SendEmail/SendRawEmail operations
- Show last send time and total sends per user
- Track verified identities and their status
- Provide sending statistics including bounces and complaints
- Generate both summary and detailed per-account reports
- Cache data for faster subsequent runs

#### Usage

The recommended way to run this tool is through make:

```bash
make ses_users_report
```

If you need to run the script directly (not recommended), ensure you've followed the [Direct Script Execution](#direct-script-execution) setup steps first.

#### Options

These options apply only when running the script directly:
- `--use-cache`: Use cached data instead of fetching new data

Note: The make target automatically handles caching for optimal performance.

#### Output

The script generates multiple reports:

1. `ses_users_summary.txt`: Overall statistics including:
   - Total SES users across accounts
   - Active users in last 30 days
   - Total sends
   - List of accounts with SES activity

2. Individual account reports (`ses_users_[account_name].txt`):
   - List of verified identities and their status
   - Per-user activity:
     * Last send time
     * Total sends
     * Recent events (last 5)
   - Account sending statistics:
     * Delivery attempts
     * Bounces
     * Complaints
     * Rejects

#### Usage

The recommended way to run this tool is through make:

```bash
make ses_identities_report
```

If you need to run the script directly (not recommended), ensure you've followed the [Direct Script Execution](#direct-script-execution) setup steps first.

#### Options

These options apply only when running the script directly:
- `--use-cache`: Use cached data instead of fetching new data

Note: The make target automatically handles caching for optimal performance.

#### Examples

1. Generate a report of SES identities (recommended):
   ```bash
   make ses_identities_report
   ```

#### Output

The script generates a single text report:

- `ses_identities_report.txt`: Comprehensive report of all SES identities across all accounts

### cove_ses_cleanup

This tool analyzes SES users and permissions across AWS accounts to identify cleanup opportunities.

#### Features

- Analyze SES usage and permissions across multiple AWS accounts
- Track user activity and access patterns
- Monitor SES sending statistics
- Identify:
  * Active SES users
  * Inactive users with SES permissions
  * Unused SES identities
  * Historical sending patterns
- Generate cleanup recommendations based on:
  * Last user activity
  * Access key usage
  * Password last used dates
  * SES sending statistics
- Cache data for faster subsequent runs

#### Usage

The recommended way to run this tool is through make:

```bash
make ses_cleanup_report
```

If you need to run the script directly (not recommended), ensure you've followed the [Direct Script Execution](#direct-script-execution) setup steps first.

#### Options

These options apply only when running the script directly:
- `--use-cache`: Use cached data instead of fetching new data

Note: The make target automatically handles caching for optimal performance.

#### Output

The script generates two files:

1. `ses_cleanup_report.txt`: Comprehensive analysis including:
   - Active Users (to keep):
     * Recent activity timestamps
     * Sending statistics
     * Account associations
   - Candidates for Removal:
     * Inactive users
     * Last activity dates
     * Permission details
     * Usage patterns

2. `ses_cleanup_cache.json`: Raw data cache including:
   - User details and permissions
   - SES statistics
   - Identity verification status
   - Access patterns

### cove_lambda_last_execution

This tool analyzes the last execution time of Lambda functions across AWS accounts.

#### Usage

The recommended way to run this tool is through make:

```bash
# Basic execution report
make lambda_last_execution_report

# Report for never-executed functions only
make lambda_last_execution_report_never_only
```

If you need to run the script directly (not recommended), ensure you've followed the [Direct Script Execution](#direct-script-execution) setup steps first.

#### Options

These options apply only when running the script directly:
- `--use-cache`: Use cached data instead of fetching new data
- `--errors-only`: Show only Lambda functions with missing permissions

Note: The make targets automatically handle caching for optimal performance.

### cove_lambda_cloudwatch_permissions

This tool checks Lambda functions' permissions for CloudWatch Logs across AWS accounts.

#### Usage

The recommended way to run this tool is through make:

```bash
# Basic permissions report
make lambda_cloudwatch_permissions_report

# Report for errors only
make lambda_cloudwatch_permissions_report_errors_only
```

If you need to run the script directly (not recommended), ensure you've followed the [Direct Script Execution](#direct-script-execution) setup steps first.

#### Options

These options apply only when running the script directly:
- `--use-cache`: Use cached data instead of fetching new data
- `--errors-only`: Show only Lambda functions with CloudWatch Logs permission issues

Note: The make targets automatically handle caching for optimal performance.

### cove_lambda_timeouts

This tool analyzes Lambda functions across AWS accounts for timeout issues by examining CloudWatch logs.

#### Features

- Analyze Lambda function timeouts across multiple AWS accounts
- Monitor function execution timeouts by examining CloudWatch logs
- Track configured timeout settings for each function
- Identify functions experiencing timeout issues
- Generate detailed reports including:
  * Timestamp of timeout events
  * Error messages and context
  * Configured timeout values
- Cache data for faster subsequent runs

#### Usage

The recommended way to run this tool is through make:

```bash
# Generate full timeout analysis report
make lambda_timeouts_report

# Generate report for functions with timeout issues only
make lambda_timeouts_report_issues_only
```

If you need to run the script directly (not recommended), ensure you've followed the [Direct Script Execution](#direct-script-execution) setup steps first.

#### Options

These options apply only when running the script directly:
- `--use-cache`: Use cached data instead of fetching new data
- `--timeouts-only`: Show only Lambda functions with timeout issues

Note: The make targets automatically handle caching for optimal performance.

#### Output

The script generates reports based on the options used:

1. `lambda_timeout_audit.txt` (default): Comprehensive report including:
   - All Lambda functions across accounts
   - Configured timeout values
   - Timeout status (OK or TIMEOUT DETECTED)
   - Detailed timeout events with timestamps
   - Summary statistics

2. `lambda_timeout_issues.txt` (with --timeouts-only): Focused report including:
   - Only functions that experienced timeouts
   - Detailed timeout events and timestamps
   - Error messages and context

### cove_lambda_expired_never_run

This tool identifies Lambda functions with expired runtimes that have never been executed.

#### Usage

The recommended way to run this tool is through make:

```bash
make lambda_expired_never_run_report
```

If you need to run the script directly (not recommended), ensure you've followed the [Direct Script Execution](#direct-script-execution) setup steps first.

Note: The make target automatically handles environment setup for optimal execution.

### cove_route53_zones

This tool collects and reports on Route53 hosted zones across AWS accounts.

#### Features

- List all hosted zones across multiple AWS accounts
- Group zones by TLD or AWS account
- Show internal/external zone status
- Sort zones alphabetically within groups
- Cache data for faster subsequent runs

#### Usage

The recommended way to run this tool is through make:

```bash
# Group by TLD (default)
make route53_zones_report

# Group by AWS account
make route53_zones_report GROUP_BY=account
```

If you need to run the script directly (not recommended), ensure you've followed the [Direct Script Execution](#direct-script-execution) setup steps first.

#### Options

These options apply only when running the script directly:
- `--use-cache`: Use cached data instead of fetching new data
- `--group-by`: Group zones by 'tld' (default) or 'account'

### cove_route53_health

This tool performs comprehensive health checks on Route53 and domain registrations across AWS accounts.

#### Features

- Analyze Route53 hosted zones and domain registrations across multiple AWS accounts
- Check domain expiration and configuration
- Verify DNS security settings
- Optional health check verification for A/AAAA records
- Optional CAA record verification
- Generate categorized summary and detailed reports
- Cache data for faster subsequent runs

#### Usage

The recommended way to run this tool is through make:

```bash
# Basic health report
make route53_health_report

# Full report with health checks and CAA verification
make route53_health_report_full
```

If you need to run the script directly (not recommended), ensure you've followed the [Direct Script Execution](#direct-script-execution) setup steps first.

#### Options

These options apply only when running the script directly:
- `--use-cache`: Use cached data instead of fetching new data
- `--check-health`: Enable health check verification for A/AAAA and alias records
- `--check-caa`: Enable CAA record verification

Note: The make targets automatically handle caching and appropriate verification options.

#### Examples

1. Generate basic health report (recommended):
   ```bash
   make route53_health_report
   ```

2. Generate comprehensive report with all verifications:
   ```bash
   make route53_health_report_full
   ```

#### Output

The script generates two reports:

1. `route53_summary_report.txt`: Categorized overview of findings including:
   - Domain expiration issues
   - Domain security issues
   - Domain configuration issues
   - Zone configuration issues
   - Health check issues (if enabled)
   - Errors

2. `route53_detailed_report.txt`: Comprehensive configuration details including:
   - Hosted zone configurations
   - Domain registration details
   - Health check configurations
   - Full list of identified issues

### cove_waf_v1_detector

This tool detects AWS WAF V1 resources across all accounts in preparation for the end of support in September 2025.

#### Features

- Detect WAF V1 resources across multiple AWS accounts
- Identify both global WAF and regional WAF resources
- Find WebACLs, Rules, and associated resources
- Track resource details including names, IDs, and configurations
- Generate comprehensive reports including:
  * Account-by-account inventory of WAF V1 resources
  * Regional distribution of WAF resources
  * Associated resources for regional WebACLs
  * Summary statistics and totals
- Detailed error reporting for troubleshooting

#### Usage

The recommended way to run this tool is through make:

```bash
make waf_v1_detector_report
```

If you need to run the script directly (not recommended), ensure you've followed the [Direct Script Execution](#direct-script-execution) setup steps first.

#### Options

These options apply only when running the script directly:
- `--output FILENAME`: Specify a custom output file name (default: waf_v1_inventory.txt)
- `--json`: Also output detailed JSON results for further analysis

Note: The make target automatically handles the appropriate options.

#### Output

The script generates a comprehensive report:

1. `waf_v1_inventory.txt`: Complete inventory including:
   - Account-by-account listing of WAF V1 resources
   - Global WAF resources (WebACLs and Rules)
   - Regional WAF resources organized by region
   - Associated resources for each WebACL
   - Summary statistics including:
     * Total accounts scanned
     * Accounts using WAF V1
     * Global and regional WebACL counts
     * Global and regional Rule counts
   - Any exceptions or errors encountered
   - Failed role assumptions

2. Optional JSON output (with --json):
   - `waf_v1_inventory_detailed.json`: Raw data for programmatic analysis

### cove_s3_bucket

This tool generates an inventory of S3 buckets across AWS accounts.

#### Features

- List all S3 buckets across multiple AWS accounts
- Filter results for a specific bucket name
- Generate detailed reports including:
  * Bucket names organized by account
  * Account IDs and names
  * Failed role assumptions
  * Exceptions and errors
- Support for focused bucket analysis
- Comprehensive error reporting

#### Usage

The recommended way to run this tool is through make:

```bash
# Generate full bucket inventory
make s3_bucket_inventory

# Get details for a specific bucket
make s3_bucket_details BUCKET_NAME=your-bucket-name
```

If you need to run the script directly (not recommended), ensure you've followed the [Direct Script Execution](#direct-script-execution) setup steps first.

#### Options

These options apply only when running the script directly:
- `--bucket BUCKET_NAME`: Restrict output to a specific bucket name

Note: The make targets automatically handle the appropriate options.

#### Output

The script generates reports based on the options used:

1. `s3_bucket_inventory.txt` (default): Complete inventory including:
   - All buckets organized by AWS account
   - Account IDs and names
   - Any exceptions or errors encountered
   - Failed role assumptions

2. `s3_bucket_details_[bucket_name].txt` (with --bucket): Focused report including:
   - Specific bucket details across accounts
   - Account ownership information
   - Any related errors or issues

### cove_route53_resolver_rules

This tool analyzes Route53 resolver rules and endpoints across AWS accounts.

#### Features

- Analyze resolver rules and endpoints across multiple AWS accounts
- Track resolver endpoint configurations and health
- Monitor resolver rule configurations and VPC associations
- Track shared rules across accounts
- Generate summary and detailed reports
- Cache data for faster subsequent runs

#### Usage

The recommended way to run this tool is through make:

```bash
make route53_resolver_rules_report
```

If you need to run the script directly (not recommended), ensure you've followed the [Direct Script Execution](#direct-script-execution) setup steps first.

#### Options

These options apply only when running the script directly:
- `--use-cache`: Use cached data instead of fetching new data

Note: The make target automatically handles caching for optimal performance.

#### Output

The script generates two reports:

1. `route53_resolver_rules_summary.txt`: Overview report including:
   - Total counts of rules, endpoints, and shared rules
   - Distribution of rule types
   - Domain patterns analysis
   - Shared rules by source account
   - Issues overview

2. `route53_resolver_rules_detailed.txt`: Comprehensive configuration details including:
   - Resolver endpoint configurations
   - Rule configurations with target IPs
   - VPC associations
   - Shared rule details
   - Full list of identified issues

### cove_ip_finder

This tool searches for a specific IPv4 address across all AWS accounts and resources.

#### Features

- Search for IPv4 addresses across multiple AWS accounts
- Identify resources using the IP address:
  * EC2 instances (primary/secondary, private/public IPs)
  * Network interfaces
  * Load balancers (ALB/NLB/CLB)
  * VPC Endpoints (Interface, Gateway, Gateway Load Balancer)
  * RDS instances
- Provides detailed information about found resources including:
  * Account information
  * VPC ID
  * Resource type and ID
  * Resource name (where available)
  * IP type (private/public/endpoint)

#### Usage

The recommended way to run this tool is through make:

```bash
make ip_finder IP_ADDRESS=x.x.x.x
```

For example:
```bash
make ip_finder IP_ADDRESS=10.0.1.100
```

If you need to run the script directly (not recommended), ensure you've followed the [Direct Script Execution](#direct-script-execution) setup steps first:

```bash
./bin/cove_ip_finder 10.0.1.100
```

#### Output

The tool provides console output with detailed information about any resources found using the specified IP address:

- Account information (name and ID)
- Resource details:
  * Type (EC2, ENI, Load Balancer, VPC Endpoint, RDS)
  * Resource ID
  * Resource Name
  * VPC ID
  * IP Type
  * Additional context (e.g., instance attachments, VPC endpoint services)

### cove_iam_users

This tool analyzes IAM users across all AWS accounts, including the main account.

#### Features

- Analyze IAM users across all accounts (including main account)
- Generate detailed reports for each account containing:
  * User details (creation date, last access times)
  * Password and access key usage information
  * MFA status
  * User tags
  * Detailed permissions:
    - AWS managed policies (shown by name)
    - Customer managed policies (full YAML format)
    - Inline policies (full YAML format)
- Generate summary statistics:
  * Total users across accounts
  * MFA adoption rates
  * Access key usage statistics
- Cache data for faster subsequent runs

#### Usage

The recommended way to run this tool is through make:

```bash
make iam_users_report
```

If you need to run the script directly (not recommended), ensure you've followed the [Direct Script Execution](#direct-script-execution) setup steps first.

#### Options

These options apply only when running the script directly:
- `--use-cache`: Use cached data instead of fetching new data

Note: The make target automatically handles caching for optimal performance.

#### Output

The script generates multiple reports:

1. `iam_users_summary.txt`: Overall statistics including:
   - Total user count across accounts
   - MFA adoption statistics
   - Access key usage statistics
   - List of accounts with IAM users

2. Individual account reports (`iam_users_[account_name].txt`):
   - Detailed user information
   - Creation dates and last access times
   - Password and access key usage
   - MFA status
   - User tags
   - Complete permission details:
     * AWS managed policies listed by name
     * Customer managed policies in YAML format
     * Inline policies in YAML format

### cove_cloudtrail_status

This tool analyzes CloudTrail configurations and status across AWS accounts.

#### Features

- Analyze CloudTrail configurations across multiple AWS accounts
- Track trail status and logging activity
- Monitor trail configurations including:
  * Multi-region trail settings
  * Organization trail settings
  * S3 bucket destinations
  * KMS encryption status
  * Log file validation
  * CloudWatch Logs integration
  * Custom event selectors
- Generate comprehensive reports on:
  * Overall CloudTrail status and statistics
  * Detailed configurations per account
  * Potential issues and security concerns
- Cache data for faster subsequent runs

#### Usage

The recommended way to run this tool is through make:

```bash
make cloudtrail_status_report
```

If you need to run the script directly (not recommended), ensure you've followed the [Direct Script Execution](#direct-script-execution) setup steps first.

#### Options

These options apply only when running the script directly:
- `--use-cache`: Use cached data instead of fetching new data

Note: The make target automatically handles caching for optimal performance.

#### Output

The script generates three reports:

1. `cloudtrail_audit_summary.txt`: Overview report including:
   - Total accounts analyzed
   - Accounts with CloudTrail configured
   - Statistics on multi-region trails
   - Organization trail usage
   - Encryption and logging status
   - CloudWatch Logs integration stats

2. `cloudtrail_audit_detailed.txt`: Comprehensive configuration details including:
   - Per-account trail configurations
   - Trail settings and status
   - Logging destinations
   - Security features enabled
   - Latest delivery times and errors

3. `cloudtrail_audit_issues.txt`: Security and configuration concerns including:
   - Inactive trails
   - Missing log file validation
   - Unencrypted trails
   - Missing logging destinations
   - Delivery errors

## Reports

To generate all reports, including the new ones, run:

```
make reports
```

Available report categories:

1. lambda_runtimes_report: Reports on Lambda function runtimes across all accounts.
2. lambda_last_execution_report: Provides information about the last execution time of Lambda functions.
3. lambda_last_execution_report_never_only: Reports only on Lambda functions that have never been executed.
4. codebuild_runtimes_report: Reports on CodeBuild project runtimes.
5. lambda_cloudwatch_permissions_report: Checks Lambda functions' permissions for CloudWatch Logs.
6. lambda_cloudwatch_permissions_report_errors_only: Reports only on Lambda functions with CloudWatch Logs permission issues.
7. lambda_expired_never_run_report: Identifies Lambda functions with expired runtimes that have never been executed.
8. route53_health_report: Reports on Route53 and domain registration health across accounts.
9. route53_health_report_full: Comprehensive Route53 health report including health checks and CAA verification.
10. route53_resolver_rules_report: Reports on Route53 resolver rules and endpoints across accounts.
11. iam_users_report: Analyzes IAM users across all accounts, including permissions and security settings.
12. waf_v1_detector_report: Detects WAF V1 resources across all accounts (end of support Sep 2025).

To generate a specific report, use:

```
make <report_name>
```

For example:
```
make lambda_expired_never_run_report
```

All reports are generated in the `reports` directory within `cove-tools`. This directory location can be configured in the Makefile by modifying the `REPORTS_DIR` variable.

## AWS Profile

The Makefile in this directory sets the AWS_PROFILE to the main account before executing the report generators. This is necessary because the reports have to be run from within the main account, as cove assumes a role in all accounts that can only be assumed from the main account.

## Requirements

- Python 3.x
- boto3
- botocove
- jinja2 (for cove_lambda_runtimes HTML reports)

## Installation

1. Clone the repository:
   ```
   git clone <repository-url>
   ```

2. Navigate to the project directory:
   ```
   cd Infra/hc-tools/cove-tools
   ```

3. Install the required dependencies:
   ```
   make .direnv
   ```

## cove_lambda_runtimes

### Options

- `--full-report`: Generate a full report of all runtimes (default is only deprecated/end-of-support)
- `--language LANGUAGE`: Filter results by programming language (e.g., python, nodejs) or specific version (e.g., python3.7)
- `--html`: Generate an HTML report instead of console output
- `--output FILENAME`: Specify the output file for the HTML report (only applicable with --html)

### Examples

1. Generate a report of deprecated and end-of-support runtimes:
   ```
   ./bin/cove_lambda_runtimes
   ```

2. Generate a full report of all runtimes:
   ```
   ./bin/cove_lambda_runtimes --full-report
   ```

3. Filter results for Python runtimes:
   ```
   ./bin/cove_lambda_runtimes --language python
   ```

4. Generate an HTML report for all NodeJS runtimes:
   ```
   ./bin/cove_lambda_runtimes --full-report --language nodejs --html --output nodejs_report.html
   ```

5. Generate an HTML report for deprecated Python 3.7 runtimes:
   ```
   ./bin/cove_lambda_runtimes --language python3.7 --html --output python37_deprecated.html
   ```

### Output

#### Console Output
The console output provides a summary of Lambda runtimes across all accounts, followed by a breakdown by account. It includes information such as runtime name, function count, lifecycle state, and end of support date.

#### HTML Report
The HTML report provides a more detailed and visually structured output. It organizes the information by account and then by runtime, displaying:

- Account name
- Runtime details (name, count, state, end of support date)
- List of functions using each runtime, along with their associated stacks

The HTML report is styled for easy readability and can be viewed in any web browser.

## Note

Ensure you have the necessary AWS permissions to access Lambda, CodeBuild, SES, Route53, EC2, RDS, CloudFormation, and WAF information across the accounts you're analyzing.

## Contributing

Contributions to improve the tools are welcome. Please submit pull requests with any enhancements or bug fixes.