semvec/pypi-packages huggingface dataset:
magnus-extension-datastore-s3 0.2.0
Magnus extension for S3 as run log store
License: Apache-2.0
Requires Python: >=3.8,<4.0

# S3 Run Log store

This package is an extension to [magnus](https://github.com/AstraZeneca/magnus-core).

## Provides 
Provides the functionality to use S3 buckets as run log stores.

## Installation instructions

```pip install magnus_extension_datastore_s3```

## Set up required to use the extension

Access to AWS environment either via:

- AWS profile, generally stored in ~/.aws/credentials
- AWS credentials available as environment variables

If you are using environmental variables for AWS credentials, please set:

- AWS_ACCESS_KEY_ID: AWS access key
- AWS_SECRET_ACCESS_KEY: AWS access secret
- AWS_SESSION_TOKEN: The session token, useful to assume other roles

A S3 bucket that you want to use to store the run log store.

Since the run log would be modified frequently as part of the run, we recommend 
[switching off versioning](https://docs.aws.amazon.com/AmazonS3/latest/userguide/Versioning.html) of files in
the S3 bucket.


## Config parameters

The full configuration of S3 Run log store are:

```yaml
run_log:
  type: s3
  config:
    s3_bucket: # Bucket name, required
    region: # Region if we are using
    aws_profile: #The profile to use or default
    use_credentials: # Defaults to False
```

### **s3_bucket**:

The s3 bucket you want to use for storing the run logs.

Please note that S3 run log store is not concurrent friendly and therefore might be inconsistent 
if you are running parallel steps.

The below parameters are inherited from AWS Configuration.

### **aws_profile**:

The profile to use for acquiring boto3 sessions. 

Defaults to None, which is used if its role based access or in case of credentials present as environmental variables.

### **region**:

The region to use for acquiring boto3 sessions.

Defaults to *eu-west-1*.


### **aws_credentials_file**:

The file containing the aws credentials.

Defaults to ```~/.aws/credentials```.

### **use_credentials**:

Set it to ```True``` to provide AWS credentials via environmental variables.

Defaults to ```False```.

### ***role_arn***:

The role to assume after getting the boto3 session.

**This is required if you are using ```use_credentials```.**