Setting up the development environment on Google Virtual Machine
Published: 2023-03-30 06:33:48

<p>I'm participating in this year's cohort of the <a target="_blank" href="https://github.com/DataTalksClub/data-engineering-zoomcamp">Data Engineering Zoomcamp 2023.</a> This is a community-led, free data engineering course of about 8 weeks. In this blog, I'll summarise the steps to configure a Google Virtual Machine to make it ready for the rest of the course.</p>
<h2 id="heading-configure-ssh-keys">Configure SSH Keys</h2>
<p>Generate a new SSH key with the following commands:</p>
<pre><code class="lang-bash"><span class="hljs-built_in">cd</span> ~/.ssh
ssh-keygen -t rsa -f &lt;key-file-name&gt; -C &lt;username&gt; -b 2048
</code></pre>
<p>It'll raise a prompt to enter a passphrase. You can leave it and press <code>enter</code>. If it asks for confirmation, press <code>enter</code> again. Here's an example:</p>
<p><img src="https://cdn.hashnode.com/res/hashnode/image/upload/v1676808135879/b1991e1e-17f7-48ad-9969-42b0bfa139e5.png" alt class="image--center mx-auto" /></p>
<p>This generates 2 files in the .ssh folder, one for the public (<code>gcp-blog.pub</code>) and one for the private key (<code>gcp-blog</code>).</p>
<p><img src="https://cdn.hashnode.com/res/hashnode/image/upload/v1676808454293/65b3151b-8e81-44d4-bad4-d773350341f5.png" alt class="image--center mx-auto" /></p>
<p>Next, upload the public key to GCP with the following steps:</p>
<ul>
<li>Open the <code>gcp-blog.pub</code> file and copy its contents. Or you can use the <code>cat</code> command to display the contents in the terminal.</li>
</ul>
<p><img src="https://cdn.hashnode.com/res/hashnode/image/upload/v1676959147606/be6e4c24-d37d-4648-8db1-25e3a5af226d.png" alt class="image--center mx-auto" /></p>
<ul>
<li>Go to the Google Cloud console &gt; Compute Engine &gt; Settings &gt; Metadata.</li>
</ul>
<p><img src="https://cdn.hashnode.com/res/hashnode/image/upload/v1676958736525/5de62633-306d-41a4-85d4-e26848decba8.png" alt class="image--center mx-auto" /></p>
<ul>
<li>Click on SSH Keys &gt; Add SSH Keys</li>
</ul>
<p><img src="https://cdn.hashnode.com/res/hashnode/image/upload/v1676958954494/e1839785-46aa-47d7-9521-c53beafff57a.png" alt class="image--center mx-auto" /></p>
<ul>
<li>Paste the contents of the public key that you copied previously on the text box and click Save.</li>
</ul>
<p><img src="https://cdn.hashnode.com/res/hashnode/image/upload/v1676959617038/92a7383a-6381-4664-9fc1-2d1973961641.png" alt class="image--center mx-auto" /></p>
<p>Now, you can connect to your Google VMs using the following command:</p>
<pre><code class="lang-bash">ssh -i &lt;PATH_TO_PRIVATE_KEY&gt; &lt;USERNAME&gt;@&lt;EXTERNAL_IP&gt;
</code></pre>
<h2 id="heading-create-a-virtual-machine">Create a Virtual Machine</h2>
<p>To set up a Virtual Machine:</p>
<ul>
<li>Go to Compute Engine &gt; VM Instances</li>
</ul>
<p><img src="https://cdn.hashnode.com/res/hashnode/image/upload/v1676960349912/be0d9107-6b5c-4cb1-b42c-267f86b23ac2.png" alt class="image--center mx-auto" /></p>
<ul>
<li>Click on Create Instance.</li>
</ul>
<p><img src="https://cdn.hashnode.com/res/hashnode/image/upload/v1676960443427/43d7004b-5764-4695-b70d-e5b8aa2b85cd.png" alt class="image--center mx-auto" /></p>
<ul>
<li>Populate the configurations for the VM with the following details (<em>Name and Region can be as per your preference</em>):</li>
</ul>
<p><img src="https://cdn.hashnode.com/res/hashnode/image/upload/v1676960829175/4d599ca8-df67-4c2b-b8eb-09c9a1c393bc.png" alt class="image--center mx-auto" /></p>
<ul>
<li>Next, change the boot disk with the following configurations:</li>
</ul>
<p><img src="https://cdn.hashnode.com/res/hashnode/image/upload/v1676964375425/41f1d4b8-dbc0-4151-ae37-5dff3699d368.png" alt class="image--center mx-auto" /></p>
<ul>
<li>Leave the rest of the configurations to default values and click Create.</li>
</ul>
<p>This will spin up a virtual machine instance for you. In order to ssh into this instance, run the following command:</p>
<pre><code class="lang-bash">ssh -i &lt;PATH_TO_PRIVATE_KEY&gt; &lt;USERNAME&gt;@&lt;EXTERNAL_IP&gt;
</code></pre>
<p>Here's an example on my system:</p>
<p><img src="https://cdn.hashnode.com/res/hashnode/image/upload/v1676964776339/6ae41758-2a8b-41b4-bbc0-a1341b49718c.png" alt class="image--center mx-auto" /></p>
<p>You can also configure an ssh alias, which is a convenient way to store the ssh details in a config file. You can follow my blog on this and set up your alias to easily connect with a VM.</p>
<div class="embed-wrapper"><div class="embed-loading"><div class="loadingRow"></div><div class="loadingRow"></div></div><a class="embed-card" href="https://itsadityagupta.hashnode.dev/ssh-simplified-aliasing-credentials-with-config-files">https://itsadityagupta.hashnode.dev/ssh-simplified-aliasing-credentials-with-config-files</a></div>
<p> </p>
<p>I have created an alias for the VM by the name <code>dezoomcamp</code>. Here's the new command to ssh:</p>
<pre><code class="lang-bash">ssh dezoomcamp
</code></pre>
<p><img src="https://cdn.hashnode.com/res/hashnode/image/upload/v1676965031665/26148bb9-5801-49be-bb46-672cc72f79ba.png" alt class="image--center mx-auto" /></p>
<blockquote>
<p>If you want to connect to a VM using any other options, please go through the official documentation on <a target="_blank" href="https://cloud.google.com/compute/docs/connect/standard-ssh#connect_to_vms"><strong>Connecting to VMs</strong></a><strong>.</strong></p>
</blockquote>
<h2 id="heading-configure-the-virtual-machine">Configure the Virtual Machine</h2>
<p>Now that you have a Virtual Machine running and a way to ssh into it and run Linux commands, let's start with installing the requirements of the course to make it ready for development.</p>
<h4 id="heading-installing-anaconda">Installing Anaconda</h4>
<ul>
<li>Visit <a target="_blank" href="https://www.anaconda.com/products/distribution">Anaconda's website</a> and copy the link to Linux 64-Bit Installer.</li>
</ul>
<p><img src="https://cdn.hashnode.com/res/hashnode/image/upload/v1676965555810/fde6028a-9b3f-49c1-a9a2-6bba3b37fd21.png" alt class="image--center mx-auto" /></p>
<ul>
<li><p>Download the file in the VM:</p>
</li>
<li><pre><code class="lang-bash">      wget https://repo.anaconda.com/archive/Anaconda3-2022.10-Linux-x86_64.sh
</code></pre>
</li>
<li><p>Run the downloaded file:</p>
</li>
<li><pre><code class="lang-bash">      bash Anaconda3-2022.10-Linux-x86_64.sh
</code></pre>
</li>
<li><p>Keep pressing Enter to scroll down and enter <code>yes</code> to accept the license terms.</p>
</li>
<li><p>Press Enter to confirm the default location.</p>
</li>
<li><p>Enter <code>yes</code> to run the <code>conda init</code> when asked.</p>
</li>
<li><p>After the installation is complete, run the command <code>source .bashrc</code> to apply the changes to the <code>.bashrc</code> file. Alternatively, you can log out of the session using the <code>logout</code> command and then ssh back in for the changes to take effect.</p>
</li>
</ul>
<p><img src="https://cdn.hashnode.com/res/hashnode/image/upload/v1676966299230/ef36655c-fcfe-4096-bee4-acb02e48a2f2.png" alt class="image--center mx-auto" /></p>
<p>You'll notice the anaconda environment name in the shell prompt once the changes are applied. Also, from the above image it is confirmed that python is also installed.</p>
<h4 id="heading-installing-docker">Installing Docker</h4>
<p>Run the following commands:</p>
<pre><code class="lang-bash">sudo apt-get update
sudo apt-get upgrade
sudo apt-get install docker.io
</code></pre>
<p>This will install docker but you'll not be able to run it without <code>sudo</code>. It'll throw a permission denied error:</p>
<p><img src="https://cdn.hashnode.com/res/hashnode/image/upload/v1676966712431/e49b7047-f3e2-4a51-9b95-25a9b2cb099b.png" alt class="image--center mx-auto" /></p>
<p>To run docker without <code>sudo</code>, run the following commands:</p>
<pre><code class="lang-bash">sudo groupadd docker
sudo gpasswd -a <span class="hljs-variable">$USER</span> docker
sudo service docker restart
</code></pre>
<p>(Refer to <a target="_blank" href="https://github.com/sindresorhus/guides/blob/main/docker-without-sudo.md">this link</a> for an explanation of the above commands.)</p>
<p>Log out of the ssh session and log back in to re-evaluate the group memberships and try running the <code>docker run hello-world</code> command.</p>
<p><img src="https://cdn.hashnode.com/res/hashnode/image/upload/v1676967224382/7bd906e1-1790-47b1-bda1-8558bc33c11e.png" alt class="image--center mx-auto" /></p>
<p>This time it works!</p>
<h4 id="heading-installing-docker-compose">Installing docker-compose</h4>
<ul>
<li><p>Go to the docker-compose <a target="_blank" href="https://github.com/docker/compose">GitHub repo</a> and under the latest release, find the "<code>docker-compose-linux-x86_64</code>" asset and copy its link.</p>
</li>
<li><p>Create a new <code>bin</code> folder in VM and download the asset into it:</p>
</li>
<li><pre><code class="lang-bash">      mkdir bin
      <span class="hljs-built_in">cd</span> bin
      wget https://github.com/docker/compose/releases/download/v2.16.0/docker-compose-linux-x86_64 -O docker-compose
</code></pre>
</li>
<li><p>This will create a new file <code>docker-compose</code> in the <code>bin</code> folder.</p>
</li>
<li><p>Make this file executable:</p>
</li>
<li><pre><code class="lang-bash">      chmod +x docker-compose
</code></pre>
</li>
<li><p>Now, add the bin folder to the path:</p>
<ul>
<li><p>Go to the home directory: <code>cd</code></p>
</li>
<li><p>Open the .bashrc file: <code>nano .bashrc</code></p>
</li>
<li><p>Paste the following to the end of the <code>.bashrc</code> file:</p>
</li>
<li><pre><code class="lang-bash">      <span class="hljs-built_in">export</span> PATH=<span class="hljs-string">"<span class="hljs-variable">${HOME}</span>/bin:<span class="hljs-variable">${PATH}</span>"</span>
</code></pre>
</li>
<li><p>Press <code>Ctrl+O</code> &gt; <code>enter</code> &gt; <code>Ctrl+X</code></p>
</li>
<li><p>Run <code>source .bashrc</code> for the changes to take effect.</p>
</li>
</ul>
</li>
</ul>
<p>Now, you have docker-compose installed as well.</p>
<p><img src="https://cdn.hashnode.com/res/hashnode/image/upload/v1676968180159/71ca4022-fbb0-486b-baff-c8a063b71159.png" alt class="image--center mx-auto" /></p>
<h4 id="heading-installing-pgcli">Installing Pgcli</h4>
<p>Pgcli is used to connect to the Postgres database and execute queries. It can be installed using the following command:</p>
<pre><code class="lang-bash">pip install pgcli
</code></pre>
<p>Once the installation is complete, you can connect to a Postgres database using the command:</p>
<pre><code class="lang-bash">pgcli -h &lt;hostname&gt; -u &lt;username&gt; -d &lt;database-name&gt;
</code></pre>
<p>It will prompt for the password afterwards.</p>
<h4 id="heading-installing-terraform">Installing Terraform</h4>
<ul>
<li><p>Go to terraform's <a target="_blank" href="https://developer.hashicorp.com/terraform/downloads">installation website</a>.</p>
</li>
<li><p>Copy the link for Linux's AMD64 file.</p>
</li>
<li><p><img src="https://cdn.hashnode.com/res/hashnode/image/upload/v1677043960701/f9352f96-1ed2-4385-a370-d5dac99c9955.png" alt class="image--center mx-auto" /></p>
<p>  Go to the bin folder created previously and download the file using <code>wget</code>:</p>
</li>
<li><pre><code class="lang-bash">    <span class="hljs-built_in">cd</span> ~/bin
    wget https://releases.hashicorp.com/terraform/1.3.9/terraform_1.3.9_linux_amd64.zip
</code></pre>
</li>
<li><p>To unzip this file, install the unzip package:</p>
</li>
<li><pre><code class="lang-bash">    sudo apt-get install unzip
</code></pre>
</li>
<li><p>Now, unzip the file:</p>
</li>
<li><pre><code class="lang-bash">    unzip terraform_1.3.9_linux_amd64.zip
</code></pre>
</li>
<li><p>There will be a <code>terraform</code> executable file extracted. You can delete the zip file.</p>
</li>
<li><p>Since this file is in the bin folder and the bin folder is in the path, everything is set up.</p>
</li>
<li><p>Try running the following command to verify terraform's installation:</p>
</li>
<li><pre><code class="lang-bash">    terraform --version
</code></pre>
<p>  <img src="https://cdn.hashnode.com/res/hashnode/image/upload/v1677045726326/8039e644-41bd-4374-970a-6776ad6dbe91.png" alt class="image--center mx-auto" /></p>
</li>
</ul>
<h4 id="heading-creating-a-service-account">Creating a service account</h4>
<ul>
<li>Go to IAM and Admin &gt; Service Accounts</li>
</ul>
<p><img src="https://cdn.hashnode.com/res/hashnode/image/upload/v1677046425534/dca26177-3aaf-4423-bb7f-3c4c6a91caba.png" alt class="image--center mx-auto" /></p>
<ul>
<li><p>Click on the "Create Service Account" button at the top and provide the service account name.</p>
</li>
<li><p>Assign the following roles to this service account:</p>
</li>
</ul>
<p><img src="https://cdn.hashnode.com/res/hashnode/image/upload/v1677047523030/7c8ce8d5-be46-4c1d-83de-e77dea1789fe.png" alt class="image--center mx-auto" /></p>
<ul>
<li><p>Click Done. The service account is now created.</p>
</li>
<li><p>To generate keys for this service account, click on the 3-dot menu and then on "manage keys"</p>
</li>
</ul>
<p><img src="https://cdn.hashnode.com/res/hashnode/image/upload/v1677047814762/1d1e9382-ac2c-4f7e-acd0-8a8a2a2a1553.png" alt class="image--center mx-auto" /></p>
<ul>
<li><p>On the following page, click on Add Key &gt; Create new key&gt; JSON format. Click Create.</p>
</li>
<li><p>This will download a JSON file. This is to be uploaded to the VM using SFTP.</p>
</li>
<li><p>Navigate to the path of the downloaded file on the terminal and open an SFTP connection by running the command: <code>sftp &lt;ssh-alias-name&gt;.</code></p>
</li>
<li><p>Next, create a folder <code>.gcp</code> and upload the credentials file into it by running the following commands:</p>
</li>
<li><pre><code class="lang-bash">    mkdir .gcp
    put &lt;credentials-file-name&gt; .gcp/
</code></pre>
</li>
<li><p>This will upload the file to VM. Here's a screenshot from my system:</p>
</li>
<li><p><img src="https://cdn.hashnode.com/res/hashnode/image/upload/v1677048420681/6ed1273b-17fd-4fa4-b108-8fb4825e46cc.png" alt class="image--center mx-auto" /></p>
</li>
</ul>
<h4 id="heading-authenticate-gcp-using-the-service-account-credentials">Authenticate GCP using the service account credentials</h4>
<p>To authenticate GCP, we need to set an environment variable <code>$GOOGLE_APPLICATION_CREDENTIALS</code> to point to the service account JSON file.</p>
<pre><code class="lang-bash"><span class="hljs-built_in">export</span> GOOGLE_APPLICATION_CREDENTIALS=/home/aditya/.gcp/&lt;credentials-filename&gt;.json
</code></pre>
<p>Next, authenticate GCP using the following command:</p>
<pre><code class="lang-bash">gcloud auth activate-service-account --key-file <span class="hljs-variable">$GOOGLE_APPLICATION_CREDENTIALS</span>
</code></pre>
<p>Here's my example:</p>
<p><img src="https://cdn.hashnode.com/res/hashnode/image/upload/v1677048891324/d452eb5c-ba02-4da8-b3e1-fc8f220706b7.png" alt class="image--center mx-auto" /></p>
<h4 id="heading-installing-pyspark">Installing Pyspark</h4>
<p>Instructions to install spark on Linux are well documented on the course's GitHub repository.</p>
<p>You can follow the below 2 files in the mentioned order, to install Pyspark:</p>
<ol>
<li><p><a target="_blank" href="https://github.com/DataTalksClub/data-engineering-zoomcamp/blob/main/week_5_batch_processing/setup/linux.md"><code>linux.md</code></a></p>
</li>
<li><p><a target="_blank" href="https://github.com/DataTalksClub/data-engineering-zoomcamp/blob/main/week_5_batch_processing/setup/pyspark.md"><code>pyspark.md</code></a></p>
</li>
</ol>
<blockquote>
<p>Tip: You can write all the environment variables mentioned in the above 2 files to <code>.bashrc</code> followed by the <code>source .bashrc</code> command to make the variables permanent. Otherwise they'll need to be setup everytime you restart your VM.</p>
</blockquote>
<h4 id="heading-cloning-the-course-repo">Cloning the course repo</h4>
<p>Clone the <a target="_blank" href="https://github.com/DataTalksClub/data-engineering-zoomcamp">course repo</a> into the home directory. Your final folder structure should look something like this:</p>
<p><img src="https://cdn.hashnode.com/res/hashnode/image/upload/v1677050146025/d9ddf857-5671-4c69-9bad-d9e4c1621149.png" alt class="image--center mx-auto" /></p>
<h4 id="heading-open-a-remote-connection-from-visual-studio-code">Open a Remote Connection from Visual Studio Code</h4>
<p>Install the "Remote-SSH" extension.</p>
<p><img src="https://cdn.hashnode.com/res/hashnode/image/upload/v1677049700312/6b32e4ce-5380-4787-bb12-23d1ed8fef67.png" alt class="image--center mx-auto" /></p>
<p>Press <code>F1</code> and select "<code>Remote SSH: Connect to Host</code>". It will show you the hosts configured on your system.</p>
<p>Select the one associated with your Virtual Machine.</p>
<p>Open the course repo folder and you're all set to start your development!</p>
<hr />
<h2 id="heading-conclusion">Conclusion</h2>
<p>You can also refer to the course video for setting up the development environment <a target="_blank" href="https://www.youtube.com/watch?v=ae-CV2KfoN0&amp;list=PL3MmuxUbc_hJed7dXYoJw8DoCuVHhGEQb&amp;index=13">here</a>. <mark>Make sure to stop your VM once you're done with the setup or your development.</mark></p>
<p>Thanks for reading!</p>
</br><p style="color:#757CF9;">This article is published w/ <a target="_blank" href="https://scattr.io?ref=dev">Scattr  ↗️</a></p>