How to Manage Google Cloud Compute Engine with Python
Tags: python, googlecloud, tutorial
Published: 2023-09-08 09:47:46

Compute Engine is an IaaS offering by Google Cloud. It is a powerful platform that can be used to create and manage virtual machines. However, managing Compute Engine can be a complex task. In this blog post, I will show you how to use Python to automate the management of Compute Engine instances.

## Setting up your environment

Make sure you have python and pip installed.

### Installation

**Mac/Linux**

```bash
pip install virtualenv
virtualenv <your-env>
source <your-env>/bin/activate
<your-env>/bin/pip install google-cloud-compute
```

**Windows**

```bash
pip install virtualenv
virtualenv <your-env>
<your-env>\Scripts\activate
<your-env>\Scripts\pip.exe install google-cloud-compute
```

### Authentication

For authentication let's create a service account.

1. IAM & Admin --&gt; Service Accounts
    
2. Click on CREATE SERVICE ACCOUNT.
    
![Create Service Account](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/yggqbwymhnxdqdhypuc8.png)

     
3. Add a Service account name and Service account ID.
    
    ![Service account](https://cdn.hashnode.com/res/hashnode/image/upload/v1694104179772/dd3b8d3b-8f34-4723-89c5-afdd4e212dec.png)
    
4. Click on CREATE AND CONTINUE.
    
    ![Create](https://cdn.hashnode.com/res/hashnode/image/upload/v1694104218300/78d1b2ad-d70f-4ced-9513-0a219d5b9cdc.png)
    
5. Add Role of Compute Admin.
    
6. Click on DONE.
    

Now let's generate a JSON key which we can use in our local environment.

1. Click on the service account name you just created.
    
2. Click on the KEYS option.
    
    ![Keys](https://cdn.hashnode.com/res/hashnode/image/upload/v1694104875559/e93a05fa-eeb7-4a98-972a-7320195d97bf.png )
    
3. ADD KEY --&gt; Create new key
    
4. Select JSON and click on CREATE.
    
5. Download and Save the key.
    

Now that we have authentications sorted let's look at some code!

## Creating a new VM instance

```python
from google.cloud import compute_v1
import os

# Using os we are storing the service account json key in the variable
os.environ["GOOGLE_APPLICATION_CREDENTIALS"]= "key.json"

INSTANCE_NAME = 'instance-name'
MACHINE_TYPE = 'projects/project-id/zones/us-west4-b/machineTypes/e2-medium' 
SUBNETWORK = 'projects/project-id/regions/us-west4/subnetworks/default'
SOURCE_IMAGE = 'projects/project-id/global/images/ubuntu-2004-focal-v20230831'
NETWORK_INTERFACE = {
    'subnetwork':SUBNETWORK,
    'access_configs': [
        {
            'name':'External NAT'
        }
    ]
}

compute_client = compute_v1.InstancesClient()

config = {
    'name' : INSTANCE_NAME,
    'machine_type' : MACHINE_TYPE,
    'disks': [
        {
            'boot': True,
            'auto_delete': True,
            'initialize_params': {
                'source_image': SOURCE_IMAGE,
            }
        }
    ],

    'network_interfaces' : [NETWORK_INTERFACE]
}

print("Creating instace.....")
operation = compute_client.insert(
    project='project-id',
    zone='us-west4-b',
    instance_resource=config
)

operation.result()

print(f'Created VM Instance:{INSTANCE_NAME}')
```

**Imported Modules :**

* [`google.cloud`](http://google.cloud)`.compute_v1`: This module provides the client library for the Google Cloud Compute Engine API.
    
* `os`: This module provides access to operating system functionality.
    

This is how the code works:

1. Sets the environment variable `GOOGLE_APPLICATION_CREDENTIALS` to the path of the JSON file that contains the Google Cloud service account credentials.
    
2. Defines the following variables:
    
    * `INSTANCE_NAME`: The name of the instance to create.
        
    * `MACHINE_TYPE`: The machine type of the instance to create.
        
    * `SUBNETWORK`: The subnetwork of the instance to create.
        
    * `SOURCE_IMAGE`: The image to use for the instance.
        
    * `NETWORK_INTERFACE`: The network interface of the instance to create.
        
3. Creates an instance of the `InstancesClient` class. This class provides methods for managing Google Compute Engine instances.
    
4. Creates a dictionary called `config`. This dictionary contains the configuration for the instance to create.
    
5. Calls the `insert()` method of the `InstancesClient` class. This method creates the instance.
    
6. Waits for the operation to complete.
    
7. Prints the name of the instance that was created.
    

If you are not sure about arguments like `MACHINE_TYPE` or `SOURCE_IMAGE` then you can refer to the Google Cloud Console.

1. Compute Engine --&gt; VM instances
    
2. Click on CREATE INSTANCE.
    
3. Choose the arguments according to you.
    

Once done click on the EQUIVALENT CODE option in the upper right corner and choose REST

![Code](https://cdn.hashnode.com/res/hashnode/image/upload/v1694106656367/c6b3c833-f2d1-4fd9-9136-1c899f6acd02.png)

Here you can see all the available arguments and you can choose as per your requirements.

## List all the VM instances

```python
from google.cloud import compute_v1
from collections import defaultdict
import os

os.environ["GOOGLE_APPLICATION_CREDENTIALS"]= "key.json"

instace_client = compute_v1.InstancesClient()
request = compute_v1.AggregatedListInstancesRequest()
request.project = 'project-id'

agg_list = instace_client.aggregated_list(request=request)

all_instances = defaultdict(list)
print("Instances found:")
for zone, response in agg_list:
        if response.instances:
            all_instances[zone].extend(response.instances)
            print(f" {zone}:")
            for instance in response.instances:
                print(f" - {instance.name} ({instance.machine_type}) {instance.status}")
```

**Imported Modules :**

* `collections.defaultdict`: This module provides a defaultdict data structure, which is a dictionary where the default value is set to a specified value.
    

This is how the code works:

1. Sets the environment variable `GOOGLE_APPLICATION_CREDENTIALS` to the path of the JSON file that contains the Google Cloud service account credentials.
    
2. Creates an instance of the `InstancesClient` class. This class provides methods for managing Google Compute Engine instances.
    
3. Creates an instance of the `AggregatedListInstancesRequest` class. This class represents a request to list Google Compute Engine instances.
    
4. Sets the project ID of the project that contains the instances to list.
    
5. Calls the `aggregated_list()` method of the `InstancesClient` class. This method lists the instances in the specified project.
    
6. Creates a dictionary called `all_instances`. This dictionary will be used to store the instances that are found.
    
7. Iterates over the response from the `aggregated_list()` method.
    
8. For each zone in the response, adds the instances in that zone to the `all_instances` dictionary.
    
9. Prints the name, machine type, and status of each instance in the `all_instances` dictionary.
    

![output](https://cdn.hashnode.com/res/hashnode/image/upload/v1694107474189/d6707a62-5a47-4da3-b5c0-219455a11547.png)

This how the output looks like. I have printed the instance name, machine type and status but you can print every attribute of the instance from the `all_instances` dictionary.

## Start and stop a VM instance

```python
from google.cloud import compute_v1
import os

os.environ["GOOGLE_APPLICATION_CREDENTIALS"]= "key.json"

compute_client = compute_v1.InstancesClient()

operation = compute_client.stop(
    project='project-id',
    zone='us-west4-b',
    instance='python-instance'
)

operation.result()
print("Instace Stopped!")
```

This is straightforward. All you have to provide is project-id, zone and instance name.

1. Creates an instance of the `InstancesClient` class. This class provides methods for managing Google Compute Engine instances.
    
2. Calls the `stop()` method of the `InstancesClient` class. This method stops the instance.  
    

The same applies for starting an instance. Rather than stop you have to use the start method rest is the same.

```python
from google.cloud import compute_v1
import os

os.environ["GOOGLE_APPLICATION_CREDENTIALS"]= "key.json"

compute_client = compute_v1.InstancesClient()

operation = compute_client.start(
    project='project-id',
    zone='us-west4-b',
    instance='python-instance'
)

operation.result()
print("Instace Started!")
```

## Delete a VM Instance

```python
from google.cloud import compute_v1
import os

os.environ["GOOGLE_APPLICATION_CREDENTIALS"]= "key.json"

compute_client = compute_v1.InstancesClient()

operation = compute_client.delete(
    project='project-id',
    zone='us-west4-b',
    instance='instance-name'
)

operation.result()
print("Instace deleted successfully!")
```

Deleting also has the same logic as starting or stopping.

1. Creates an instance of the `InstancesClient` class. This class provides methods for managing Google Compute Engine instances.
    
2. Calls the `delete()` method of the `InstancesClient` class. This method deletes the instance.
    

All the code mentioned in this blog is available on GitHub

{% embed https://github.com/pratikkalein/GCP-Python no-readme %} 

That's all about how you can manage VM instances using python. Let me know your thoughts or if I missed something in the comments.

Thank you for reading!

Connect With Me :

{% embed https://pratik.linkb.org/ %}