GCP Professional Data Engineer Guide - September 2020
Tags: googlecloud, career, datascience, machinelearning
Published: 2020-10-06 07:01:11

<!-- wp:paragraph -->
<p>I have recently recalled <strong>my first experience with GCP</strong>. It was in London, shortly before the 2012 Olympics, in an <strong>online gaming project</strong>, initially thought for <em>AWS</em>, that was migrated to App Engine -  <em><strong>PAAS platform that would evolve to the current GCP</strong></em>.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p><strong>My initial impression was good,</strong> although the platform <strong>imposed a number of development limitations</strong>, which would be reduced later with the release of <strong><em>App Engine Flexible</em></strong>.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Coinciding with the launch of <em><strong>Tensor Flow</strong></em> as an Open Source framework in 2015, I was lucky enough to <strong>attend a workshop on neural networks</strong> - given by one of the AI scientists from <em>Google Seattle</em> - where I had my second experience with the platform. I was very surprised by the <strong>simplicity of configuration and deployment</strong>, the <strong>NoOps concept and a Machine Learning / AI offering</strong>, without competition at the time.</p>
<!-- /wp:paragraph -->

<!-- wp:embed  -->
{% vimeo 132700334 %}
<!-- /wp:embed -->

<!-- wp:paragraph {"fontSize":"small"} -->
<p class="has-small-font-size">Do Androids Dream of Electric Sheep? Philip K. Dick would have "hallucinated" with the electrical dreams of neural networks - powered by Tensor Flow.</p>
<!-- /wp:paragraph -->

<!-- wp:heading -->
<h2>Exam</h2>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>The <strong>structure of the exam</strong> is the usual one in GCP exams: <em><strong>2 hours and 50 questions</strong></em>, with a format  directed towards scenario-type questions, mixing <strong>questions of high difficulty</strong> with simpler ones of <strong>medium-low difficulty</strong>.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>In general, <strong>to choose the correct answer</strong>, you <em><strong>have to apply technical and business criteria</strong></em>. Therefore, it is necessary a deep knowledge of the services from the technological point of view, as well as skill / experience to apply the business criteria in a contextual way, depending on the question, type of environment, sector, application, etc .. .</p>
<!-- /wp:paragraph -->

<!-- wp:image {"id":295,"width":698,"height":600,"sizeSlug":"large"} -->
<figure class="wp-block-image size-large is-resized"><img src="https://metanube.files.wordpress.com/2020/08/bd-data-lake.png?w=698" alt="" class="wp-image-295" width="698" height="600"/><figcaption>Image #1, Data Lake, the ubiquitous architecture - Image owned by GCP</figcaption></figure>
<!-- /wp:image -->

<!-- wp:paragraph -->
<p>We can group the relevant services according to the states (and substates) of the data cycle:</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p><em><strong>Management, Storage, Transformation and Analysis.</strong></em></p>
<!-- /wp:paragraph -->

<!-- wp:list -->
<ul><li><strong><em>Ingestion Batch</em></strong> / <strong><em>Data Lake</em></strong>: Cloud Storage.</li><li><em><strong>Ingestion</strong></em><strong><em> Streaming</em></strong>: Kafka, Pub/Sub, Computing Services, Cloud IoT Core.</li><li><em><strong>Migrations:</strong> </em>Transfer Appliance, Transfer Service, Interconnect, gsutil.</li><li><em><strong>Transformation</strong></em>s: Dataflow, Dataproc, Cloud Dataprep, Hadoop, Apache Beam.</li><li><strong><em>Computing: </em></strong>Kubernetes Engine, Compute Instances, Cloud Functions, App Engine.</li><li><strong><em>Storage</em></strong>: Cloud SQL, Cloud Spanner, Datastore / Firebase, BigQuery, BigTable, HBase, MongoDB, Cassandra.</li><li><strong><em>Cache</em></strong>: Cloud Memorystore, Redis.</li><li><em><strong>Analysis / Data Operations:</strong> </em>BigQuery, Cloud Datalab, Data Studio, DataPrep, Cloud Composer, Apache Airflow.</li><li><strong><em>Machine Learning:</em></strong> AI Platform, BigQueryML, Cloud AutoML, Tensor Flow, Cloud Text-to-Speech API, Cloud Speech-to-Text, Cloud Vision API, Cloud Video AI, Translations, Recommendations API, Cloud Inference API,  Natural Language,  DialogFlow,  Spark MLib.</li><li><strong><em>IoT</em></strong>: Cloud IoT Core, Cloud IoT Edge. </li><li><strong><em>Security &amp; encryption</em></strong>: IAM, Roles, Encryption, KMS, Data Prevention API, Compliance ... </li><li><em><strong>Opera</strong>t<strong>ions:</strong></em>  Kubeflow, AI Platform, Cloud Deployment Manager ... </li><li><strong><em>Monitorization:</em></strong> Cloud Stackdriver Logging, Stackdriver Monitoring.</li><li><strong><em>Optimization:</em></strong> Cost control, Autoscaling, Preemptive instances ...</li></ul>
<!-- /wp:list -->

<!-- wp:heading {"level":3} -->
<h3>Pre-requisites and recommendations</h3>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p><strong>At this level of certification, the questions do not refer, in general, to a single topic</strong>. That is, a question from the Analytics domain may require more or less advanced knowledge of <em>Computing, Security, Networking or DevOps</em> to be able to solve it successfully. I´d recommend having the <em>GCP Associate Cloud Engineer</em> certification or have equivalent knowledge.</p>
<!-- /wp:paragraph -->

<!-- wp:list -->
<ul><li><strong><em>GCP experience at the architectural level.</em></strong> The exam is focused, in part, on the architecture solution, design and deployment of data pipelines; selection of technologies to solve business problems, and to a lesser extent development. I´d recommend studying as many <a href="https://cloudblog.withgoogle.com/products/application-development/13-popular-application-architectures-for-google-cloud/amp/">reference architectures </a>as possible, such as the ones I show in this guide.</li><li><strong><em>GCP experience at the development level.</em></strong> Although no explicit programming questions appeared in my question set, or in the mock test, the exam requires technical knowledge of services and APIS: <em>SQL, Python, REST, algorithms, Map-Reduce, Spark, Apache Beam (Dataflow) </em>…</li><li><em><strong>GCP experience at Security level.</strong></em> Domain that appears transversally in all certifications - I´d  recommend knowledge at the level of <em>Associate Engineer</em>.</li><li><strong><em>GCP experience at Networking level.</em></strong> Another domain that appears transversely -  I´d recommend knowledge at the level of Associate Engineer.</li><li><strong><em>Knowledge of Data Analytics.</em></strong> It's a no-brainer, but some domain knowledge is essential. Otherwise, I´d recommend studying books like <em>“Data Analytics with Hadoop”</em> or taking courses like Specialized Program: <em><a href="https://www.coursera.org/specializations/gcp-data-machine-learning">Data Engineering, Big Data and ML on Google Cloud in Coursera</a>.</em> Likewise, practicing with laboratories or pet projects is essential to obtain some practical experience.</li><li><strong><em>Knowledge of the Hadoop - Spark ecosystem</em></strong>. Connected with the previous point. High-level knowledge of the ecosystem is necessary: <em>Map Reduce, Spark, Hive, Hdfs, Pig</em> …</li><li><strong><em>Knowledge of Machine Learning and IoT. </em></strong>Advanced knowledge in <em>Data Science and Machine Learning</em> is essential, apart from specific knowledge of GCP products. There are questions exclusively about this domain - at the level of certifications like <em>AWS Machine Learning </em>or higher. IoT appears on the exam in a lighter form, but it is essential to know the architecture and services of reference.</li><li><strong><em>DevOps experience. </em></strong>Concepts such as CI / CD, infrastructure or configuration as code, are of great importance today, and this is reflected in the exam, although they do not have a great specific weight.</li></ul>
<!-- /wp:list -->

<!-- wp:paragraph -->
<p></p>
<!-- /wp:paragraph -->

<!-- wp:heading {"level":3} -->
<h3>Standard questions</h3>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>Representative question of the level of difficulty of the <a href="https://cloud.google.com/certification/practice-exam/data-engineer">exam</a>.</p>
<!-- /wp:paragraph -->

<!-- wp:image {"id":351,"sizeSlug":"large"} -->
<figure class="wp-block-image size-large"><img src="https://metanube.files.wordpress.com/2020/08/captura-de-pantalla-2020-08-29-a-las-16.19.35.png?w=1024" alt="" class="wp-image-351"/><figcaption>Image property of GCP</figcaption></figure>
<!-- /wp:image -->

<!-- wp:paragraph -->
<p>Practical migration scenario question, that includes cloud services and the <em>Hadoop ecosystem</em>, as well as concepts from the <em>Analytics domain</em>.</p>
<!-- /wp:paragraph -->

<!-- wp:heading {"level":4} -->
<h4><strong>Services to study in detail</strong></h4>
<!-- /wp:heading -->

<!-- wp:image {"id":305,"sizeSlug":"large"} -->
<figure class="wp-block-image size-large"><img src="https://metanube.files.wordpress.com/2020/08/retail-rt-inventory.png?w=996" alt="" class="wp-image-305"/><figcaption>Image #2 - property of GCP</figcaption></figure>
<!-- /wp:image -->

<!-- wp:list -->
<ul><li><strong><em>Cloud Storage </em></strong>- Core service that appears consistently in all certifications, and is central in the  <em><strong>Data Lake</strong></em> systems. I´d recommend its study in detail at an architectural level - see Image 1 -, configurations according to the data temperature, and as an integration / storage element between the different services</li><li><strong><em>BigQuery</em></strong> - Core service in the <em>Analytics GCP</em> domain as a BI and storage element. Extremely important in the exam, so have to be studied in detail: architecture, configuration, backups, export / import, streaming, batch, security, partitioning, sharding, projects, datasets, views, integration with other services, cost, queries and optimization SQL (legacy and standard) at table levels, keys …</li><li><strong><em>Pub / Sub</em></strong> - Core service as an element of ingestion and integration. Its in-depth study is highly recommended: use cases, architecture, configuration, API, security and integration with other services (eg <em>Dataflow, Cloud Storage</em>) - Kafka's native cloud mirror service.</li><li><strong><em>Dataflow</em></strong> - Core service in the <em>Analytics GCP domain</em> as a process and transformation element. Implementation based on <em>Apache Beam</em> that is necessary to know at a high level and pipeline design. Use cases, architecture, configuration, API and integration with other services.</li><li><em><strong>Dataproc</strong></em> - Core service in the <em>Analytics GCP domain</em> as a process and transformation element. It is a service based on Hadoop, and therefore, it is the indicated service for a migration to the cloud. In this case, not only knowledge of Dataproc is required, but also in native services: <em>Spark, HDFS, HBase, Pig</em> … use cases, architecture, configuration, import / export, reliability, optimization, cost, API and integration with other services.</li><li><strong><em>Cloud SQL, Cloud Spanner</em></strong> - Cloud native relational databases. Use cases, architecture, configuration, security, performance, reliability, cost and optimization: clusters, transactionality, disaster recovery, backups, export / import, SQL performance and optimization, tables, queries, keys and debugging. Integration with other services.</li><li><strong><em>Cloud Bigtable</em></strong> - Low latency <em>NoSQL</em> managed database, suitable for time series, IoT… ideal to replace a <em>HBase</em> installation on premise. Use cases, architecture, configuration, security, performance, reliability and optimization: clusters, CAP, backups, export / import, partitioning, performance, and optimization of tables, queries, keys. Integration with other services. </li><li><strong><em>Machine Learning</em></strong> - One of the strengths of the certification is the domain "<em>Operationalizing machine learning models"</em>. Much more dense and complex than it may seem at first, since it not only includes the operability and knowledge of the relevant GCP services; likewise, it includes the knowledge of the Data Science fundamentals: algorithm selection, optimization, metrics … The level of difficulty of the questions is variable, but comparable to that of specific certifications, such as <em>AWS Certified Machine Learning - Specialty</em>. Most important services: <em>BigQuery ML, Cloud Vision API, Cloud Video Intelligence, Cloud AutoML, Tensor Flow, Dialogflow, GPU´s, TPU´s </em>…</li><li><strong><em>Security</em></strong> - Security is a transversal concern across all domains, and appears consistently in all certifications. In this case, it appears as an independent technical topic, crosscutting concern or as a business requirement: <em>KMS, IAM, Policies, Roles, Encryption, Data Prevention API …</em></li></ul>
<!-- /wp:list -->

<!-- wp:image {"id":397,"sizeSlug":"large"} -->
<figure class="wp-block-image size-large"><img src="https://metanube.files.wordpress.com/2020/09/12_2fk8rre.max-1100x1100-1.png?w=1024" alt="" class="wp-image-397"/><figcaption>Image #3, IoT Reference Architecture - owned by GCP</figcaption></figure>
<!-- /wp:image -->

<!-- wp:paragraph -->
<p></p>
<!-- /wp:paragraph -->

<!-- wp:heading {"level":4} -->
<h4><strong>Very important services to consider</strong></h4>
<!-- /wp:heading -->

<!-- wp:list -->
<ul><li><strong><em>Networking</em></strong> - Cross-domain that can appear in the form of separate technical issues, cross cutting concerns, or as business requirements: <em>VPC, Direct Interconnect, Multi Region / Zone, Hybrid connectivity, Firewall rules, Load Balancing, Network Security, Container Networking, API Access ( private / public) …</em></li><li><strong><em>Hadoop -</em></strong> The exam covers ecosystems and third-party services like <em>Hadoop, Spark, HDFS, Hive, Pig</em> … use cases, architecture, functionality, integration and migration to GCP.</li><li><strong><em>Apache Kafka</em></strong> - Alternative service to <em>Pub / Sub</em>, so it is advisable to study it at a high level: use cases, operational characteristics, configuration, migration and integration with GCP - plugins, connectors.</li><li><strong><em>IoT</em></strong> - It can appear in various questions at the architectural level: use cases, reference architecture and integration with other services.<em> IoT core, Edge Computing.</em></li><li><strong><em>Datastore / Firebase</em></strong> - Document database. Use cases, configuration, performance, entity model, keys and index optimization, transactions, backups, export / import and integration with other services. It doesn't carry as much weight as the other data repositories.</li><li><strong><em>Cloud Memory Store / Redis</em></strong> - Structured data cache repository. Use cases, architecture, configuration, performance, reliability and optimization: clusters, backups, export / import and integration with other services.</li><li><strong><em>Cloud Dataprep </em></strong>- Use cases, console and general operation, supported formats, and Dataflow integration.</li><li><strong><em>Cloud Stackdriver</em></strong> - Use cases, monitoring and logging, both at the system and application level: <em>Cloud Stackdriver Logging, Cloud Stackdriver Monitoring, Stackdriver Agent and plugins.</em></li></ul>
<!-- /wp:list -->

<!-- wp:heading {"level":4} -->
<h4><strong>Other services</strong></h4>
<!-- /wp:heading -->

<!-- wp:list -->
<ul><li><strong><em>MongoDB, Cassandra - </em></strong><em>NoSQL</em> databases that can appear in different scenarios. Use cases, architecture and integration with other services<strong><em>.</em></strong></li><li><strong><em>Cloud Composer </em></strong>- Use cases, general operation and web console, configuration of diagram types, supported formats, import / export, integration with other services, connectors.</li><li><strong><em>Cloud Data Studio</em></strong> - Use cases, configuration, networking, security, general operation and environment, and integration with other services.</li><li><strong><em>Cloud Data Lab -</em></strong> Use cases, general operation and web console, types of diagrams, supported formats, import / export and integration with other services.</li><li><strong><em>Kubernetes  Engine - </em></strong>Use cases, architecture, clustering and integration with other services.</li><li><strong><em>Kubeflow - </em></strong>Use cases, architecture, environment configuration, Kubernetes.</li><li><strong><em>Apache Airflow</em></strong>  - Use cases, architecture and general operation.</li><li><strong><em>Cloud Functions </em></strong> -<strong><em> </em></strong>Use cases, architecture, configuration and integration with other services - such as <em>Cloud Storage and Pub / Sub, in Push / Pull mode.</em></li><li><strong><em>Compute Engine </em></strong>- Use cases, architecture, configuration, high availability, reliability and integration with other services.</li><li><strong><em>App Engine -</em></strong> Use cases, architecture and integration with other services.</li></ul>
<!-- /wp:list -->

<!-- wp:paragraph -->
<p></p>
<!-- /wp:paragraph -->

<!-- wp:heading -->
<h2>Bibliography &amp; essential resources</h2>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p><strong>Google provides</strong> a large number of <strong>resources for the preparation of this certification</strong>, in the form of courses, official guide book, documentation and mock exams. These resources are <strong>highly recommended</strong>, and in some cases, I would say essential.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>The <a href="https://www.coursera.org/learn/preparing-cloud-professional-data-engineer-exam">Certification Preparation Course</a>, contained in the <a href="https://www.coursera.org/specializations/gcp-data-machine-learning">Data Engineering Specialized Program</a>, includes an extra exam, lots of additional tips and materials and labs - using the external Qwik Labs tool.</p>
<!-- /wp:paragraph -->

<!-- wp:list -->
<ul><li><em><a href="https://cloud.google.com/certification/guides/data-engineer">GCP Certification Guide</a></em></li><li><em><a href="//cloud.google.com/docs#section-9">Google Doc</a></em><a href="//cloud.google.com/docs#section-9"><em>s</em></a></li><li><em><a href="https://cloud.google.com/certification/practice-exam/data-engineer">Practice exam</a></em></li><li><em><a href="https://www.coursera.org/learn/preparing-cloud-professional-data-engineer-exam">Readiness course</a> </em>– <em>highly recommended, includes an additional practice test.</em></li><li><em><a href="https://www.coursera.org/specializations/gcp-data-machine-learning">Data Engineering, Big Data and ML on Google Cloud</a></em></li><li><a href="https://cloudblog.withgoogle.com/products/application-development/13-popular-application-architectures-for-google-cloud/amp/"><em>Thirteen GCP Reference Architectures</em></a></li></ul>
<!-- /wp:list -->

<!-- wp:image {"id":389,"sizeSlug":"large"} -->
<figure class="wp-block-image size-large"><img src="https://metanube.files.wordpress.com/2020/08/20200827_090027.jpg?w=1024" alt="" class="wp-image-389"/><figcaption><br>Bibliography (selection) that I have used for the preparation of the certification</figcaption></figure>
<!-- /wp:image -->

<!-- wp:paragraph -->
<p>As I have previously indicated, <strong>I find the Google courses on Coursera to be excellen</strong>t, as they combine a series of <strong>short videos, reading material, labs, and test questions</strong>, thus creating <strong>a very dynamic experience</strong>. In any case, they <strong>should only be considered as a starting point</strong>, being necessary the deepening - according to experience - in each one of the domains using, for instance, <em>the excellent GCP documentation.</em></p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>But you should not limit yourself to online courses. I can't hide the fact that I love books in general, and IT books in particular. In fact, I have a huge collection of books dating back to the 80s, which at some point I will donate to a local Cervantina bookstore.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p><strong>Books provide</strong> a deeper and more <strong>dynamic experience than videos</strong>, which can be a bit monotonous if they are too long - as well as being a much more passive experience - like watching TV. The ideal is the <strong>combination of audiovisual and written media</strong>, <strong>thus creating your own learning path</strong>.</p>
<!-- /wp:paragraph -->

<!-- wp:heading {"level":3} -->
<h3>Laboratories</h3>
<!-- /wp:heading -->

<!-- wp:list -->
<ul id="block-0cc6ede6-3159-4db7-82fc-d6154af6ef29"><li><a href="https://www.qwiklabs.com/quests/43"><em>Data Science on GCP Quest</em></a></li><li><a href="https://www.qwiklabs.com/quests/25"><em>Data Engineering Quest</em></a></li></ul>
<!-- /wp:list -->

<!-- wp:image {"id":362,"sizeSlug":"large"} -->
<figure class="wp-block-image size-large"><img src="https://metanube.files.wordpress.com/2020/08/6_5xbiker.0999056719991134.max-1100x1100-1.png?w=1024" alt="" class="wp-image-362"/><figcaption>Image #4  - Data Lake based upon Cloud Storage - owned by GCP</figcaption></figure>
<!-- /wp:image -->

<!-- wp:paragraph -->
<p>Part of the job as a <strong><em>Data Engineer</em></strong> consists of creating, integrating, deploying and maintaining <strong>data pipelines</strong>, both in batch and streaming mode.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>The <a href="https://www.qwiklabs.com/quests/25">Data Engineering Quest</a> contains several labs that introduce <strong>the creation of different data transformation,  IoT, and Machine Learning pipelines</strong>, so I find them excellent exercises - and not just for certification.</p>
<!-- /wp:paragraph -->

<!-- wp:heading -->
<h2>Is it worth?</h2>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p><strong>The level of certification is advanced,</strong> and in general, it should not be the first cloud certification to obtain. It <strong>covers a large amount of material and domains</strong>, so tackling it without a certain level of prior knowledge <strong>can be quite a complex task.</strong></p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>If we compare it with the mirror certification on the AWS platform, it covers almost twice as much material, mainly due to the <strong>inclusion of questions about the Machine Learning / Data Science domain</strong> - which in the case of AWS have been eliminated, to be included in its own certification. Therefore, it is like taking two certifications in one.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Is it worth? of course, but not as a first certification - depending on the experience provided.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p><strong>Certifications</strong> are a good way, not only to <strong>validate knowledge externally</strong>, but to <strong>collect updated information,</strong> <strong>validate good practices and consolidate knowledge</strong> with real practical cases (or almost).</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Good luck to you all!</p>
<!-- /wp:paragraph -->