Storage Transfer Service in GCP
Published: 2023-03-30 06:34:19

<h2 id="heading-introduction">Introduction</h2>
<p>The Storage Transfer Service is a <strong>data migration service</strong> provided by Google Cloud Platform (GCP) that allows users to move large amounts of data between on-premises data centres, cloud storage solutions, and other cloud providers.</p>
<p>The Storage Transfer Service can transfer data from sources such as Amazon S3, Microsoft Azure, and other third-party storage providers, as well as from file servers and other on-premises storage systems. Users can set up transfers that are one-time or recurring, with customizable scheduling and filtering options.</p>
<p><em>The service supports various transfer options, including parallel transfer and resumable transfer, which helps to ensure data integrity and efficient data transfer. Additionally, it provides features like data validation, notification, and logging that enable users to monitor the transfer process and detect issues.</em></p>
<p>The Storage Transfer Service can be managed using the GCP Console or through APIs, and it is charged based on the amount of data transferred and the number of operations performed.</p>
<hr />
<p>In this blog post, I'll guide you through the process of creating a storage transfer service using Terraform to transfer data from one Google Cloud Storage (GCS) bucket to another. To follow along with this tutorial, you'll need to have the following prerequisites in place:</p>
<ol>
<li><p>Terraform installed on your local machine</p>
</li>
<li><p>A Google Cloud Platform (GCP) project created in the Google Cloud Console</p>
</li>
<li><p>A GCP service account with a JSON private key file and the necessary permissions</p>
</li>
</ol>
<p>If you haven't created a service account yet, check out my previous blog post on how to create one.</p>
<div class="embed-wrapper"><div class="embed-loading"><div class="loadingRow"></div><div class="loadingRow"></div></div><a class="embed-card" href="https://itsadityagupta.hashnode.dev/setting-up-a-gcp-service-account">https://itsadityagupta.hashnode.dev/setting-up-a-gcp-service-account</a></div>
<p> </p>
<h2 id="heading-assumptions">Assumptions</h2>
<p>I have a Google Cloud Storage (GCS) bucket named "<strong>gcs_capstone_dezoomcamp</strong>" that contains approximately <strong>9 gigabytes</strong> of data. My goal is to transfer all the files from this bucket to the "data" folder within a <strong>new</strong> bucket called "<strong>destination</strong>".</p>
<blockquote>
<p><strong>Note:</strong> The source bucket is already created whereas the destination bucket has to be created.</p>
</blockquote>
<h2 id="heading-initializing-a-terraform-project">Initializing a Terraform Project</h2>
<ol>
<li>Create a new directory for your Terraform project and create a new file named "<strong>main.tf</strong>". In the "<strong>main.tf</strong>" file, define the provider by specifying "google" as the provider and your project credentials.</li>
</ol>
<div class="gist-block embed-wrapper" data-gist-show-loading="false" data-id="4f30855c9a70477d662fe5a4325e81db"><div class="embed-loading"><div class="loadingRow"></div><div class="loadingRow"></div></div><a href="https://gist.github.com/Aditya-Gupta1/4f30855c9a70477d662fe5a4325e81db" class="embed-card">https://gist.github.com/Aditya-Gupta1/4f30855c9a70477d662fe5a4325e81db</a></div><p> </p>
<blockquote>
<p>Note: If the environment variable <code>GOOGLE_APPLICATION_CREDENTIALS</code> is set to the path of the service account key file, then the <code>credentials</code> fields is not necessary.</p>
</blockquote>
<p>Run the <code>terraform init</code> command to initialize the project.</p>
<h2 id="heading-create-the-destination-gcs-bucket">Create the destination GCS bucket</h2>
<p>Use the "<strong>google_storage_bucket</strong>" resource to define the destination bucket. Specify the "name" parameter to set the name of the bucket (globally unique), and the "location" parameter to set the location of the bucket.</p>
<div class="gist-block embed-wrapper" data-gist-show-loading="false" data-id="8a785b0c503bf07f4aab7a1b6e888ae1"><div class="embed-loading"><div class="loadingRow"></div><div class="loadingRow"></div></div><a href="https://gist.github.com/Aditya-Gupta1/8a785b0c503bf07f4aab7a1b6e888ae1" class="embed-card">https://gist.github.com/Aditya-Gupta1/8a785b0c503bf07f4aab7a1b6e888ae1</a></div><p> </p>
<h2 id="heading-grant-permissions-to-gcs-buckets">Grant permissions to GCS buckets</h2>
<p>Storage Transfer Service uses a <em>default service account</em> for data transfer which doesn't have required accesses by default. Hence, we have to configure access to the source GCS bucket and the destination GCS bucket before creating a data transfer job.</p>
<h3 id="heading-roles-for-source-gcs-bucket">Roles for source GCS bucket</h3>
<ol>
<li><p>One of:</p>
<ul>
<li><p><strong>Storage Object Viewer</strong> (<code>roles/storage.objectViewer</code>) if the transfer is to another Cloud Storage bucket.</p>
</li>
<li><p><strong>Storage Object Creator</strong> (<code>roles/storage.objectCreator</code>) if the transfer is to a file system.</p>
</li>
</ul>
</li>
<li><p>Plus one of the:</p>
<ul>
<li><p><strong>Storage Legacy Bucket Writer</strong> (<code>roles/storage.legacyBucketWriter</code>) if the object deletes permission is required.</p>
</li>
<li><p><strong>Storage Legacy Bucket Reader</strong> (<code>roles/storage.legacyBucketReader</code>) if the object deletes permission is not required.</p>
</li>
</ul>
</li>
</ol>
<h3 id="heading-assigning-roles-to-the-source-gcs-bucket">Assigning roles to the source GCS bucket</h3>
<p>To assign roles and permissions to the storage transfer service, it's important to first identify the default service account being used. This can be achieved by utilizing the "<strong>google_storage_transfer_project_service_account</strong>" data block.</p>
<p>Once the service account has been identified, it can be granted the necessary roles and added as a member to the relevant Google Cloud Storage (GCS) bucket. Let's look at the terraform code to assign roles to the source GCS bucket:</p>
<div class="gist-block embed-wrapper" data-gist-show-loading="false" data-id="a49dd76be109f5c242b285fc3641c792"><div class="embed-loading"><div class="loadingRow"></div><div class="loadingRow"></div></div><a href="https://gist.github.com/Aditya-Gupta1/a49dd76be109f5c242b285fc3641c792" class="embed-card">https://gist.github.com/Aditya-Gupta1/a49dd76be109f5c242b285fc3641c792</a></div><p> </p>
<p><em>Note that the</em> <code>legacyBucketWriter</code> <em>role is not assigned to the source bucket because, in the data transfer job, I'll not delete the source files.</em></p>
<h3 id="heading-roles-for-destination-gcs-bucket">Roles for destination GCS bucket</h3>
<ul>
<li><strong>Storage Legacy Bucket Writer</strong> (<code>roles/storage.legacyBucketWriter</code>)</li>
</ul>
<h3 id="heading-assigning-roles-to-the-destination-gcs-bucket">Assigning roles to the destination GCS bucket</h3>
<div class="gist-block embed-wrapper" data-gist-show-loading="false" data-id="8b6fc66fd3ff91f76befa86e254fcf93"><div class="embed-loading"><div class="loadingRow"></div><div class="loadingRow"></div></div><a href="https://gist.github.com/Aditya-Gupta1/8b6fc66fd3ff91f76befa86e254fcf93" class="embed-card">https://gist.github.com/Aditya-Gupta1/8b6fc66fd3ff91f76befa86e254fcf93</a></div><p> </p>
<h2 id="heading-define-the-data-transfer-job">Define the data transfer job</h2>
<p>Use the "<strong>google_storage_transfer_job</strong>" resource to define the data transfer job. Specify the source and destination bucket names as well as the desired transfer options.</p>
<div class="gist-block embed-wrapper" data-gist-show-loading="false" data-id="536facac85cf99eea1398f3ecb90ca64"><div class="embed-loading"><div class="loadingRow"></div><div class="loadingRow"></div></div><a href="https://gist.github.com/Aditya-Gupta1/536facac85cf99eea1398f3ecb90ca64" class="embed-card">https://gist.github.com/Aditya-Gupta1/536facac85cf99eea1398f3ecb90ca64</a></div><p> </p>
<h2 id="heading-overview">Overview</h2>
<p>Here's a complete <strong>"main.tf"</strong> file:</p>
<div class="gist-block embed-wrapper" data-gist-show-loading="false" data-id="3db7dedf4e85dd9bcc1d44632676441f"><div class="embed-loading"><div class="loadingRow"></div><div class="loadingRow"></div></div><a href="https://gist.github.com/Aditya-Gupta1/3db7dedf4e85dd9bcc1d44632676441f" class="embed-card">https://gist.github.com/Aditya-Gupta1/3db7dedf4e85dd9bcc1d44632676441f</a></div><p> </p>
<p>Now, <strong>run the command</strong> <code>terraform apply</code> and enter "yes" when prompted. This will perform the following:</p>
<ul>
<li><p>Create a destination bucket.</p>
</li>
<li><p>Assign roles to the default service account for the source and the destination bucket.</p>
</li>
<li><p>Create a data transfer job <strong><em>with a schedule as defined</em></strong>.</p>
</li>
</ul>
<blockquote>
<p>Make sure to run <code>terraform destroy</code> to destroy all the resources.</p>
</blockquote>
<h2 id="heading-conclusion">Conclusion</h2>
<p>By following the steps outlined in this blog post, you can easily define a storage transfer job and assign the necessary permissions to the source and destination buckets. This allows you to transfer data between buckets or even between different cloud storage providers.</p>
<p>More information about this can be found in the following blogs:</p>
<ul>
<li><p><a target="_blank" href="https://cloud.google.com/storage-transfer-service">https://cloud.google.com/storage-transfer-service</a></p>
</li>
<li><p><a target="_blank" href="https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/storage_transfer_job">https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/storage_transfer_job</a></p>
</li>
<li><p><a target="_blank" href="https://cloud.google.com/storage-transfer/docs/sources-and-sinks">https://cloud.google.com/storage-transfer/docs/sources-and-sinks</a></p>
</li>
</ul>
<p>I hope you found this blog post helpful and informative. I'd love to hear your thoughts on the topic or any issues you've had following along. Did I miss anything important?</p>
<p>Please leave a comment below and let me know what you think. Your feedback is valuable and can help me improve my writing and provide additional insights for other readers. Thank you for taking the time to read this post!</p>
<div class="hn-embed-widget" id="linkedin-follow"></div></br><p style="color:#757CF9;">This article is published w/ <a target="_blank" href="https://scattr.io?ref=dev">Scattr  ↗️</a></p>