enhanced-s3-storage-provider 1.7.0
A storage provider which can fetch and store media in Amazon S3.
License: Apache-2.0

Synapse S3 Storage Provider
===========================

This module can be used by synapse as a storage provider, allowing it to fetch
and store media in Amazon S3.


Usage
-----

The `s3_storage_provider.py` should be on the PYTHONPATH when starting
synapse.

Example of entry in synapse config:

```yaml
media_storage_providers:
- module: s3_storage_provider.S3StorageProviderBackend
  store_local: True
  store_remote: True
  store_synchronous: True
  config:
    bucket: <S3_BUCKET_NAME>
    # All of the below options are optional, for use with non-AWS S3-like
    # services, or to specify access tokens here instead of some external method.
    region_name: <S3_REGION_NAME>
    endpoint_url: <S3_LIKE_SERVICE_ENDPOINT_URL>
    access_key_id: <S3_ACCESS_KEY_ID>
    secret_access_key: <S3_SECRET_ACCESS_KEY>
    session_token: <S3_SESSION_TOKEN>

    # Server Side Encryption for Customer-provided keys
    #sse_customer_key: <S3_SSEC_KEY>
    # Your SSE-C algorithm is very likely AES256
    # Default is AES256.
    #sse_customer_algo: <S3_SSEC_ALGO>

    # The object storage class used when uploading files to the bucket.
    # Default is STANDARD.
    #storage_class: "STANDARD_IA"

    # Prefix for all media in bucket, can't be changed once media has been uploaded
    # Useful if sharing the bucket between Synapses
    # Blank if not provided
    #prefix: "prefix/to/files/in/bucket"

    # Connection pool configuration (v1.6.6+)
    # The maximum number of concurrent threads for S3 operations
    # Each thread manages a single S3 operation (upload/download)
    # Default is 20
    threadpool_size: 20
    
    # The maximum number of HTTP connections in the boto3 connection pool
    # Should be >= threadpool_size for optimal performance
    # Default is 50
    max_pool_connections: 50

    # Auto-restore configuration (NEW)
    # Automatically restore files from S3 when they're missing locally
    # Useful for ephemeral storage scenarios (e.g., Kubernetes emptyDir)
    auto_restore: true  # Enable/disable auto-restore (default: true)
    auto_restore_startup: true  # Restore all missing files at startup (default: true)
    auto_restore_monitor_interval: 300  # Monitor interval in seconds (default: 300)
    auto_restore_batch_size: 50  # Files to restore per batch (default: 50)
```

This module uses `boto3`, and so the credentials should be specified as
described [here](https://boto3.readthedocs.io/en/latest/guide/configuration.html#guide-configuration).

Connection Pool Configuration (v1.6.6+)
---------------------------------------

This version includes configurable connection pools to prevent connection exhaustion issues.

### Configuration Parameters

- **`threadpool_size`** (default: 20): Number of concurrent threads for S3 operations. Each thread handles one S3 upload/download operation at a time.
  
- **`max_pool_connections`** (default: 50): Size of the boto3/urllib3 HTTP connection pool. This should be equal to or greater than `threadpool_size` for optimal performance.

### Why This Matters

In high-traffic environments or when processing many media files simultaneously, the default connection pool settings can become exhausted, leading to:
- `Connection pool is full, discarding connection` warnings
- Failed S3 operations
- Pod restarts in Kubernetes environments
- Health check failures

### Tuning Guidelines

- **Low traffic** (< 10 concurrent uploads): Use defaults
- **Medium traffic** (10-30 concurrent uploads): `threadpool_size: 20`, `max_pool_connections: 50`
- **High traffic** (30+ concurrent uploads): `threadpool_size: 40`, `max_pool_connections: 100`

### Example Configuration
```yaml
media_storage_providers:
- module: s3_storage_provider.S3StorageProviderBackend
  store_local: True
  store_remote: True
  store_synchronous: True
  config:
    bucket: my-bucket
    endpoint_url: https://s3.amazonaws.com
    threadpool_size: 20
    max_pool_connections: 50
```

Auto-Restore Feature
-------------------

**NEW**: This enhanced version includes an automatic file restoration system that solves the ephemeral storage problem in containerized environments.

### Problem Solved
When using ephemeral storage (like Kubernetes `emptyDir`), container restarts cause all local media files to be lost. While files remain safely stored in S3, Synapse expects them to be available locally, causing 404 errors for media requests.

### Solution
The auto-restore system:
- **Startup Restoration**: Quickly restores all missing files when the container starts
- **Continuous Monitoring**: Periodically checks for missing files and restores them
- **Efficient Caching**: Uses an in-memory cache of S3 files to minimize API calls
- **Batch Processing**: Restores files in configurable batches to avoid overwhelming the system
- **Thread Pool**: Uses concurrent downloads for fast restoration

### Configuration Options
```yaml
auto_restore: true  # Enable auto-restore (default: true)
auto_restore_startup: true  # Restore missing files at startup (default: true) 
auto_restore_monitor_interval: 300  # Check interval in seconds (default: 5 minutes)
auto_restore_batch_size: 50  # Files per restoration batch (default: 50)
```

### How It Works
1. **At Startup**: Scans S3 bucket and restores any missing files to local storage
2. **During Operation**: Monitors `media_store_path` and `uploads_path` directories
3. **File Detection**: Compares local files with S3 inventory using efficient caching
4. **Restoration**: Downloads missing files using thread pool for parallelization
5. **Path Mapping**: Correctly maps S3 paths to local media and upload directories

This ensures your Synapse homeserver works seamlessly with ephemeral storage while maintaining the cost benefits of S3-only storage.

Regular cleanup job
-------------------

There is additionally a script at `scripts/s3_media_upload` which can be used
in a regular job to upload content to s3, then delete that from local disk.
This script can be used in combination with configuration for the storage
provider to pull media from s3, but upload it asynchronously.

Once the package is installed, the script should be run somewhat like the
following. We suggest using `tmux` or `screen` as these can take a long time
on larger servers.

`database.yaml` should contain the keys that would be passed to psycopg2 to
connect to your database. They can be found in the contents of the
`database`.`args` parameter in your homeserver.yaml.

More options are available in the command help.

```
> cd s3_media_upload
# cache.db will be created if absent. database.yaml is required to
# contain PG credentials
> ls
cache.db database.yaml
# Update cache from /path/to/media/store looking for files not used
# within 2 months
> s3_media_upload update /path/to/media/store 2m
Syncing files that haven't been accessed since: 2018-10-18 11:06:21.520602
Synced 0 new rows
100%|█████████████████████████████████████████████████████████████| 1074/1074 [00:33<00:00, 25.97files/s]
Updated 0 as deleted

> s3_media_upload upload /path/to/media/store matrix_s3_bucket_name --storage-class STANDARD_IA --delete
# prepare to wait a long time
```

Packaging and release
---------

For maintainers:

1. Update the `__version__` in setup.py. Commit. Push.
2. Create a release on GitHub for this version.
3. When published, a [GitHub action workflow](https://github.com/matrix-org/synapse-s3-storage-provider/actions/workflows/release.yml) will build the package and upload to [PyPI](https://pypi.org/project/synapse-s3-storage-provider/).