Manage private Python packages using Artifact Registry (Google Cloud)
Tags: googlecloud, artifactregistry, python
Published: 2022-03-06 05:30:06

Lately, I build a system over microservices on Google Cloud. I often use Python with App Engine, Cloud Functions and Cloud Run.

There are some private libraries in common across each microservice.
Referring to here official documents([App Engine](https://cloud.google.com/appengine/docs/standard/python3/specifying-dependencies#private_dependencies_with_artifact_registry), [Cloud Functions](https://cloud.google.com/functions/docs/writing/specifying-dependencies-python#private_dependencies_from_other_repositories)), we had deployed each microservice using the pip command `pip install -t` to copy dependencies into a local folder.

After a couple of week since introducing the above pip command operation when deploying, I realized that it was hard to control a system quality, because the above pip command was difficult to specify a dependency version each library in our team environments. So we could not feel free to update some libraries which had been used by many microservices.

In this article, I introduce how manage private Python packages using [Artifact Registry](https://cloud.google.com/artifact-registry) and solve those quality control issues.

All source code which I wrote about in this article has been uploaded below for your reference.
https://github.com/koshilife/artifact-registry-python-repo-example

# Set up tools and a virtual environment

The following are versions of gcloud, Python and direnv when I tried.
```sh
$ gcloud app update
$ gcloud --version
Google Cloud SDK 374.0.0
alpha 2022.02.22
beta 2022.02.22
core 2022.02.22
gsutil 5.6
kubectl 1.21.9

$ python3 --version
Python 3.9.10

$ direnv --version
2.28.0
```

I made a directory for an example app and set up a Python virtual environment. You need `twine` which is for publishing Python and `keyrings.google-artifactregistry-auth` which handles authentication with Artifact Registry repositories. The official guide is [here](https://cloud.google.com/artifact-registry/docs/python/store-python) for your reference. 

```sh
$ mkdir hello-artifact-registry
$ cd hello-artifact-registry

$ python3 -m venv venv
$ source venv/bin/activate
$ python3 -m pip install --upgrade pip
$ pip install --upgrade pip setuptools wheel
$ pip install twine keyrings.google-artifactregistry-auth
```

The following is a direnv file.
```sh
export GCP_PROJECT="<YOUR-GCP-PROJECT>"

# a credential file path of service account which is granted read/write for Artifact Registry
export GOOGLE_APPLICATION_CREDENTIALS="<YOUR-PATH>/google_cloud_credntials.json"
```

# Create a repository on Artifact Registry

I made a repository `artifact-example` for Python library at Tokyo region using the below gcloud command. The related official guide is [here](https://cloud.google.com/artifact-registry/docs/manage-repos
)

```sh
$ gcloud artifacts repositories create artifact-example \
    --repository-format=python \
    --location=asia-northeast1 \
    --description="Python private libraries"
```

# Upload some packages to the repository

For testing I prepared very simple python libraries which just return `foo` or `bar` string.

A directory structure of these libraries is below.

```sh
python_libs
├── artifact-example-foo
│   ├── artifact_example_foo
│   │   └── __init__.py
│   ├── requirements.txt
│   └── setup.py
└── artifact-example-bar
    ├── artifact_example_bar
    │   └── __init__.py
    ├── requirements.txt
    └── setup.py
```

artifact-example-foo (v1)
__init__.py
```python
__version__ = '0.0.1'

def foo():
    return 'foo'
```

artifact-example-bar (v1)
__init__.py
```python
__version__ = '0.0.1'

def bar():
    return 'bar'
```

I packaged as wheel format and uploaded it to the repository as the below commands.

```sh
$ cd python_libs/artifact-example-foo
$ python setup.py bdist_wheel
$ twine upload --repository-url https://asia-northeast1-python.pkg.dev/${GCP_PROJECT}/artifact-example/ dist/*

$ cd python_libs/artifact-example-bar
$ python setup.py bdist_wheel
$ twine upload --repository-url https://asia-northeast1-python.pkg.dev/${GCP_PROJECT}/artifact-example/ dist/*
```

I made additional versions each library for testing as v2. On the same way I packaged and uploaded to the repository.

artifact-example-foo (v2)
```python
__version__ = '0.0.2'

def foo():
    return 'foo'

def foo2():
    return 'foo2'
```

artifact-example-bar (v2)
```python
__version__ = '0.0.2'

def bar():
    return 'bar'

def bar2():
    return 'bar2'
```

# Use these uploaded libraries in client Python apps

I created simple Python apps (Flask) in App Engine, Cloud Functions and Cloud Run in the same Google Cloud project as the repository of Artifact Registry.

Notice: If you create in another Google Cloud project, you might need to grant a service account of Code Build. (The related official guide is [here](https://cloud.google.com/artifact-registry/docs/deploy))

## App Engine

client_apps/appengine/main.py
```python
from flask import Flask, jsonify
import artifact_example_foo as foo
import artifact_example_bar as bar

app = Flask(__name__)

@app.route('/')
def hello():
    return jsonify(dict(result='ok', foo=foo.foo(), foo_version=foo.__version__, bar=bar.bar2(), bar_version=bar.__version__))

if __name__ == '__main__':
    app.run(host='127.0.0.1', port=8080, debug=True)
```

client_apps/appengine/requirements.txt
```requirements.txt
--extra-index-url https://asia-northeast1-python.pkg.dev/<YOUR-GCP-PROJECT>/artifact-example/simple
artifact-example-foo==0.0.1
artifact-example-bar
Flask==2.0.2
```

In the requirements.txt file, a point is that the Artifact Registry URL is added into the first line and the relevant packages to include private dependencies `artifact-example-foo` and `artifact-example-bar`.

Also, the `/simple` string at the end of the repository URL is needed because of [Python Simple Repository API](https://www.python.org/dev/peps/pep-0503/).

In the dependencies body, for testing I specified `artifact-example-foo` to 0.0.1 (v1) and do not specify any versions in `artifact-example-bar`

https://cloud.google.com/appengine/docs/standard/python3/specifying-dependencies#private_dependencies_with_artifact_registry

## Cloud Functions

client_apps/functions/main.py
```python
from flask import jsonify

import artifact_example_foo as foo
import artifact_example_bar as bar

def main(request):
    return jsonify(dict(result='ok', foo=foo.foo(), foo_version=foo.__version__, bar=bar.bar2(), bar_version=bar.__version__))
```

client_apps/functions/requirements.txt
```requirements.txt
--extra-index-url https://asia-northeast1-python.pkg.dev/<YOUR-GCP-PROJECT>/artifact-example/simple
artifact-example-foo==0.0.1
artifact-example-bar
```

The method of declaring the Artifact Registry repository is the same as the above App Engine one.

https://cloud.google.com/functions/docs/writing/specifying-dependencies-python#private_dependencies_from_artifact_registry


## Cloud Run

client_apps/cloudrun/main.py
```python
import os

from flask import Flask, jsonify
import artifact_example_foo as foo
import artifact_example_bar as bar

app = Flask(__name__)

@app.route("/")
def main():
    return jsonify(dict(result='ok', foo=foo.foo(), foo_version=foo.__version__, bar=bar.bar2(), bar_version=bar.__version__))

if __name__ == "__main__":
    app.run(debug=True, host="0.0.0.0", port=int(os.environ.get("PORT", 8080)))
```

client_apps/cloudrun/requirements.txt
```requirements.txt
--extra-index-url https://asia-northeast1-python.pkg.dev/<YOUR-GCP-PROJECT>/artifact-example/simple
artifact-example-foo==0.0.1
artifact-example-bar
Flask==2.0.2
gunicorn==20.1.0
```

The method of declaring the Artifact Registry repository is the same as the above too.

client_apps/cloudrun/Dockerfile
```Dockerfile
# refs: https://cloud.google.com/run/docs/quickstarts/build-and-deploy/deploy-python-service

FROM python:3.10-slim

ENV PYTHONUNBUFFERED True
ENV GOOGLE_APPLICATION_CREDENTIALS "key.json"

ENV APP_HOME /app
WORKDIR $APP_HOME
COPY . ./

RUN pip install --upgrade pip
RUN pip install keyrings.google-artifactregistry-auth
RUN pip install --no-cache-dir -r requirements.txt

CMD exec gunicorn --bind :$PORT --workers 1 --threads 8 --timeout 0 main:app
```

I needed to enables to access the Artifact Registry repository while installing dependencies in Docker, so the following command allowed to copy the service account credentials.

```
ENV GOOGLE_APPLICATION_CREDENTIALS "key.json"
```

[The deploy script](https://github.com/koshilife/artifact-registry-python-repo-example/blob/main/client_apps/deploy_memo.sh) is for your reference.


## The Response of these client apps

The following JSON is the response of theses client apps.
You can see that the library foo's version is used 0.0.1 (v1) and the library bar's version is used 0.0.2 (v2) as expected.


```json
{
    "bar": "bar2",
    "bar_version": "0.0.2",
    "foo": "foo",
    "foo_version": "0.0.1",
    "result": "ok"
}
```

# Conclusion

By using Artifact Registry I could specify a version each library, so I would improve to manage python packages and to control a system quality finely than current the pip command `pip install -t`.

# Other references

- [Using a private repo on Artifact Registry in Google Cloud Functions](https://cloud.google.com/blog/topics/developers-practitioners/using-private-repo-artifact-registry-google-cloud-functions)
- [Managing Python packages - Artifact Registry](https://cloud.google.com/artifact-registry/docs/python/manage-packages)