graceapi 1.8.1
A package to call the Grace API from Python
License: MIT
Requires Python: <4.0,>=3.10

# GraceAPI

[![pipeline status](https://gitlab.com/2021AI/data-science/tools/graceapi/badges/main/pipeline.svg)](https://gitlab.com/2021AI/data-science/tools/graceapi/-/commits/main)
[![coverage report](https://gitlab.com/2021AI/data-science/tools/graceapi/badges/main/coverage.svg)](https://gitlab.com/2021AI/data-science/tools/graceapi/-/commits/main)
[![Latest Release](https://gitlab.com/2021AI/data-science/tools/graceapi/-/badges/release.svg)](https://gitlab.com/2021AI/data-science/tools/graceapi/-/releases)

## Compatibility

- Version 1.3.x is only compatible with **GRACE 1.45**
- Version 1.4.x is only compatible with **GRACE 1.47**
- Version 1.5.x is only compatible with **GRACE 1.48**
- Version 1.6.x is only compatible with **GRACE 1.49**
- Version 1.7.x is only compatible with **GRACE 1.50** and **GRACE 1.51**

> The package automatically checks for compatibility with the GRACE version you are using.

## Installation

You can install the package very easily by following these steps:

Install the graceapi with **pip**
> `pip install graceapi --extra-index-url https://pypi.internal.2021.services/simple`

## Usage

> Note that this is an example use. Please read the docstrings for more information about the API classes.
> More detailed examples can be found in the [examples](examples) folder and in [tests/integration](tests/integration)

You can import the package and use it as follows:

```python
from graceapi import ModelAPI, ModelImage, ModelDeployment

### Choose one authentication method. The recommended approach is with pgrace

from graceapi import PgraceAuth as Auth
auth = Auth()

# OR the old version:
# from graceapi import BasicAuth as Auth
# auth = Auth(
#     domain='grace_domain (e.g. ds.2021.services)',
#     username='grace_username',
#     password='grace_password',
# )

# Create an API object, in this example we use ModelAPI
api = ModelAPI(
    domain='grace_domain (e.g. ds.2021.services)',
    project='project_name (e.g. cphteam)',
    auth=auth
)

# List all images (Can be useful to retrieve image_id for deletion)
print(api.get_images())

# Define the image payload
image_payload = ModelImage(
    source_dir='/home/repo/path/to/model', # Without your user path i.e `/home/sso/repo` becomes `/home/repo`
    display_name='display_name',
    snapshot_name='snapshotname',
    snapshot_version='x.x.x',
    image_name='grace/python3.10-fastapi:1.7',
)

# Define the deployment payload
deployment_payload = ModelDeployment(
    deployment_name='deployment_name',
    snapshot_name='snapshot',
    snapshot_version='x.x.x',
)
```

### Using tools: Uploader

It is recommended to use the Uploader class to manage file uploads to the platform.

> If you are developing in a GRACE Workspace, ignore this tool as your files are already
> available in the platform.

Example of using the Uploader class:

```python
from pathlib import Path
from graceapi import PlatformAPI, PgraceAuth
from graceapi.tools import Uploader

# Authentication
auth = PgraceAuth()

# Initialize the PlatformAPI
platform_api = PlatformAPI(
    domain="ds.2021.services",  # platform domain, change to yours
    project="cphteam",  # project name, change to yours
    auth=auth,
)

# Initialize the Uploader
uploader = Uploader(
    platform_api=platform_api,
    username="sso"  # change to your username
)

# Define the source and target directories
source_dir = Path("../tests/data")
target_dir = Path(".graceapi_upload_dir")  # Default value, optional

# Upload files
uploader.upload_files(source=source_dir, target=target_dir)
```

You can then use your uploaded files in the platform. Like:

```python
image_payload = ModelImage(
    source_dir="/home/.graceapi_upload_dir/path/to/model",
    ...
)
```

This example demonstrates the basic usage of the Uploader class to upload files to the platform. Make sure to replace the values with your actual configuration.

### Using tools: Deployer

It is recommended to use the `Deployer` class to manage images and deployments.

Example of using the `Deployer` class:

```python
from graceapi.tools import Deployer
from graceapi import ModelAPI

# Initialize he App api
api= ModelAPI (
    domain="ds.2021.services",  # domain, change to yours
    project="cphteam",  # project name, change to yours
    auth=auth,
)

# Initialize the Deployer
deployer = Deployer(api)

# Create the image
image_response = deployer.create_image(image_payload, delete_existing=True)
deployer.wait_for_image(image_response['id'], should_stream_logs=True)

# Deploy image when finished
deployment_response = deployer.create_deployment(deployment_payload, delete_existing=True)
deployer.wait_for_deployment(deployment_response['id'])
```

### Direct API Usage

If you want to do it manually, you can use the API classes directly. However, this can
lead to a lot of boilerplate code to check for existing images, deleting them, waiting
for models to be deployed, etc.

Examples:

```python
# Create a new image, but will fail if image already exists
img_resp = api.create_image(image_payload)

# Will fail here because image is not ready
deploy_resp = api.create_deployment(deployment_payload)

# Delete a deployment
api.delete_deployment(deployment_id=deploy_resp['deployment_id'])

# Delete an image
api.delete_image(image_id=img_resp['id'])
```

## Features

This is a sample package for how to use Grace programatically.
It utilizes the following subdomains of the grace endpoint: `images`, `pipelines`, `services`, `platform`, and `assessments`

### Data Models

To ensure type safety and ease-of-use we implement pydantic _data models_ that help
us validate, transform, and serialize data.

Reference the `examples/Deployer.ipynb` notebook for usage.

> Currently, we have only implemented data models as input for image and deployment methods.

#### Available data models

You can import data models with:

`from graceapi.data_models import <data_model>`

##### Images

- `AppImage`
- `ModelImage`

##### Deployments

- `AppDeployment`
- `ModelDeployment`

Advanced deployments may require:

- `Deployment`
- `Resources`
- `ScalingConfig`

### Tools

#### `Deployer` Class

- The `Deployer` class aids in creating images and deployments to GRACE and manages retries if an item already exists.
- It offers methods to get an existing image, delete an image, and create a new image.
- It offers methods to get an existing deployment, delete a deployment, and create a new deployment.
- Users can delete existing images and deployments.
- Users can wait for images and deployments to be created.
- Integrated logging to provide informative messages and warnings during the deployment process.
- **Search For Image**: Retrieves the ID of an existing image based on snapshot name and version.
- Can stream logs from image and deployments

#### `Uploader` Class

- The `Uploader` class facilitates the uploading of files and management of SSH keys on the GRACE platform.
- It offers methods to upload files from a local directory to a remote directory on the platform.
- Users can retrieve the current workspace information.
- Users can retrieve the image associated with the current workspace.
- Users can retrieve a workspace image by its name.
- It provides functionality to upload an SSH key to the platform.
- Users can create an SSH key on the platform.
- It handles the creation of an SSH key when an existing key is found.
- Users can verify the connection to the platform.
- It includes a method to execute the SCP command to transfer files.
- Integrated logging to provide informative messages and warnings during the upload process.

### API Classes

#### `ImagesAPI` (extended by `ModelAPI` and `AppAPI`)

- **Create Image**: Allows the creation of model or application images with detailed parameters.
- **Delete Image**: Provides the ability to delete model or application images using their ID.
- **List Images**: Enables listing all the available model or application images.
- **List Base Images**: List all of the available Grace Base images.

#### `ModelAPI` & `AppAPI`

- **Deploy Image**: Facilitates the deployment of a model or application image.
- **Undeploy Image**: Allows for the undeployment of a model or application image.
- **List Deployed Services**: Lists all the currently deployed models or applications.
- [`ModelAPI`] **Get Existing Model**: Retrieves the full image object given an image id.
- **All of the above**: Inherits all the methods from `ImagesAPI`.

#### `PipelineAPI`

- **Trigger Pipeline**: Initiates the execution of a pipeline.
- **Create Pipeline**: Allows for the creation of a new pipeline.
- **Delete Pipeline**: Provides the ability to delete a pipeline.
- **List Pipelines**: Lists all the available pipelines.

#### `ProjectAssessmentAPI`

- **Get All Local Assessments**: Lists all the available local assessments.
- **Get Assessment**: Retrieves a specific assessment by name.
- **Get Assessment Answers**: Retrieves the answers for a specific assessment.
- **Post New Assessment Answers**: Posts new answers to a specific assessment.

#### `PlatformAssessmentAPI`

- **Get All Assessments**: Get all assessments in the platform

#### `PlatformAPI`

- **Get Installed Images**: Retrieves the installed images on the platform, filtered by workspaces, models, and pipelines.
- **Get Project Users**: Retrieves the users assigned to a specific project.
- **Get User Workspace**: Retrieves the workspace details for a specific user in a project.
- **Activate Workspace**: Activates the workspace for a specific user.
- **Deactivate Workspace**: Deactivates the workspace for a specific user.
- **Get SSH Keys**: Retrieves all SSH keys.
- **Delete SSH Key**: Deletes a specific SSH key.
- **Check SSH Key**: Checks the usage of a specific SSH key.
- **Add SSH Key**: Adds a new SSH key for a user.
- **Get Node Quota**: Retrieves the node quota for the project.
- **Set Open Apps**: Sets the open apps status for the project.
- **Get Project Resources**: Retrieves the resource allocation status for the project.
- **Update Project Info**: Updates the project information.
- **Get Project Info**: Retrieves the project information.
- **Get All User Tokens**: Retrieves all user tokens for a specific user.
- **Get User Token**: Retrieves a specific user token by name.

## Contribution

To contribute to the project, you can use the Makefile to setup a development environment.

```bash
make init
```

Then, you can activate a shell in the virtual environment with:

```bash
poetry shell
```

That's it :)

You can type: `make` in terminal to view a list of helpful commands.

### Versioning

We use the following semantic versioning scheme:

```bash
[major].[minor].[patch]
```

For development, it follows:

```bash
[major].[minor].[patch]-[release][build]
```

For example, at time of writing, the latest version is `1.1.6`.

To start developing on a new version, you can use the `bump2version` command:

```bash
bump2version [major|minor|patch]
```

or

```bash
make bump
```

> If you are in doubt of the semantic versioning rules, you can read more about it [here](https://semver.org/). Or just default to a `patch` version.

This will create a version: `1.1.7-dev0`. To increment your development version, use `bump2version build` which will increment `dev0` to `dev1`, `dev2`, etc.

When you create merge requests, the CI will automatically bump the version to a release for you.
A release is a version without the `dev` suffix.

### Testing

All functionality should be unittested. We use `pytest` for this. The folder structure
copies the structure of the `src/graceapi` folder inside the `tests/graceapi` folder.

The `tests` folder also includes a `scripts` folder which contains scripts that can be used to test the API end-to-end.
And a `data` folder that includes a copy of the most recent openapi specifications.

Try to make use of the `conftest.py` file for reusability and standardization across tests.

Another way to make testing easier is to utilize the `@pytest.mark.parametrize` decorator.
When using this decorator, you can pass in a list of parameters to test a function with.

For example:

```python
@pytest.mark.parametrize(
    "param1, param2, expected",
    [
        (1, 2, 3),
        (2, 3, 5),
        (3, 4, 7),
    ],
)
def test_add(param1, param2, expected):
    assert add(param1, param2) == expected
```

## Tips

We can view endpoint docs at:

- `https://apps.{domain}/docs#/`
- `https://assessments.{domain}/docs#/`
- `https://images.{domain}/docs#/`
- `https://pipelines.{domain}/docs#/`
- `https://platform.{domain}/docs#/`

We can see client secret and other information at `https://{domain}/config.js`