Data Engineering — Complete Reference Guide From A-Z [2019]
Tags: dataengineering, datascience, bigdata
Published: 2019-08-07 12:48:05

#1. Data Engineering — Fast start#

*‘A scientist can discover a new star, but he cannot make one. He would have to ask an engineer to do it for him.’* — Gordon Lindsay Glegg

Nowadays everybody wants to be a Data Scientist. What about Data Engineers? Do not take it personally, but in fact, Data Scientists are as good as the quality of data they are provided with. Since companies store their data in a variety of formats across databases and text files, the key role of Data Engineers is to build data workflows, pipelines, and ETL processes that prepare and transform data for Data Scientists making their jobs more effective.

{% youtube k22oAGEkObU %}

**More on this topic:**
[Data Engineering 101 for Dummies like Me](https://towardsdatascience.com/data-engineering-101-for-dummies-like-me-cf6b9e89c2b4)
[Data engineering: A quick and simple definition](https://www.oreilly.com/ideas/data-engineering-a-quick-and-simple-definition)
[Who Is a Data Engineer & How to Become a Data Engineer?](https://towardsdatascience.com/who-is-a-data-engineer-how-to-become-a-data-engineer-1167ddc12811)
[Data Engineers: The best friends of Data Scientists you forgot to hire.](https://uwekorn.com/2019/02/13/data-engineers-the-best-friends-of-data-scientists-you-forgot.html)
[What is a Data Engineer?](https://www.dataquest.io/blog/what-is-a-data-engineer/)


#2. Data Engineering vs. Data Science#

![data scientist vs data engineer](https://miro.medium.com/max/415/1*XfG2bNwZZdQRzZ1kvpoVxw.png)
*Image source: https://twitter.com/jessetanderson/status/1115618459725979649*


*‘Data engineers are the plumbers building a data pipeline, while data scientists are the painters and storytellers, giving meaning to an otherwise static entity.’ [Urthecast’s David Bianco notes](https://blog.panoply.io/what-is-the-difference-between-a-data-engineer-and-a-data-scientist).*

These two positions are not interchangeable. However, there is a significant overlap between Data Engineers and Data Scientists when it comes to skills and responsibilities. The main difference lies in their FOCUS on a different aspect of data utilization.


![The core competencies of data scientists and data engineers and their overlapping skills. Illustration by Jesse Anderson @jessetanderson and the Big Data Institute @bdi_oxford](https://miro.medium.com/max/700/1*UU4nXMoKBFBIWknIBa6zIQ.png)
*Image source: The core competencies of data scientists and data engineers and their overlapping skills. Illustration by Jesse Anderson @jessetanderson and the Big Data Institute @bdi_oxford*

In short, Data Engineers are focused on building infrastructure and architecture for data generation, while Data Scientists are focused on advanced mathematics and statistical analysis on that generated data.

The difference in skillsets translates into differences in languages, tools, and software that both use. Below a great overview by [DataCamp](https://www.datacamp.com/community/blog/data-scientist-vs-data-engineer) including both commercial and open-source alternatives and overlapping area.

![data camp](https://miro.medium.com/max/700/1*WEEyyerDibCg9Si2w-Yl8w.png)
*Image source: https://www.datacamp.com/community/blog/data-scientist-vs-data-engineer*

If you need another explanation between Data Engineer and Data Scientist, have a look at a widely shared AI Hierarchy of Needs by [*Monica Rogati*](https://hackernoon.com/the-ai-hierarchy-of-needs-18f111fcc007).

Data Engineering covers the first 2–3 stages, while Data Science — stages 4 and 5.

![hackernoon Monica Rogati Data Science Hierarchy of needs](https://miro.medium.com/max/700/1*GZOO60G4Y9NGMuahEe1I7A.png)
*based on https://hackernoon.com/the-ai-hierarchy-of-needs-18f111fcc007*

Data Engineers are just as important as Data Scientists but seem to be **less visible** because they tend to move away from the final analysis product. According to [Dataquest](https://www.dataquest.io/blog/what-is-a-data-engineer/), it is just like with a race car builder and race car driver.

![Data Engineer vs Data Scientist](https://miro.medium.com/max/700/1*Oy1IgzKpM-ROXWfZk7QtXg.png)

**More on this topic:**
[Data Scientist vs Data Engineer](https://www.datacamp.com/community/blog/data-scientist-vs-data-engineer)
[Data Engineer VS Data Scientist](https://towardsdatascience.com/data-engineer-vs-data-scientist-bc8dab5ac124)
[Why a data scientist is not a data engineer](https://www.oreilly.com/ideas/why-a-data-scientist-is-not-a-data-engineer)
[Data engineers vs. data scientists](https://www.oreilly.com/ideas/data-engineers-vs-data-scientists)


#3. Data Engineering Skills#

![data engineering skills](https://miro.medium.com/max/480/1*BU4HqLxNYqK1oum75BE-UQ.png)
*Image source: https://i.pinimg.com/originals/5e/23/93/5e23938b01f90a79eb73c3e69f446d64.jpg*

Data Engineers are the data professionals who prepare the ‘big data’ infrastructure to be analyzed by Data Scientists. They design, build, integrate data from various resources and then, they write complex queries on that, make sure it is easily accessible, works smoothly, and their goal is optimizing the performance of their company’s big data ecosystem.

That’s why a [good Data Engineer](https://dzone.com/articles/4-traits-of-outstanding-data-engineers) - apart from tools and languages necessary to do this job - should be first and foremost good at:

✔ Excellent Problem Solvers
✔ Multi-Disciplined
✔ Team-Oriented and Collaborative
✔ Curious and Never Stops Learning

Now, let’s have a look at **Top 8 Skills** a Data Engineer needs to acquire:


{% youtube Pym0oriyQgM %}

**1. Tools and Components of Data Architecture:** Having knowledge of building complex database systems for companies. This term also refers to processes that address the data at rest, data at motion, data sets, and how they relate to data-dependent applications and processes.

**2. Big Data Frameworks/Hadoop-based Technologies:** There are several tools in the [Hadoop Ecosystem](https://hadoop.apache.org/) which caters different purposes & professionals belonging to different backgrounds: [HDFS](https://hadoop.apache.org/docs/r1.2.1/hdfs_design.html) (Hadoop Distributed File System), [YARN](https://yarnpkg.com/en/), [MapReduce](https://hadoop.apache.org/docs/r1.2.1/mapred_tutorial.html), [PIG](https://pig.apache.org/) & [HIVE](https://hive.apache.org/), [Flume](https://flume.apache.org/) & [Sqoop](https://sqoop.apache.org/), [ZooKeeper](https://zookeeper.apache.org/), [Oozie](https://oozie.apache.org/).

![data engineering skills](https://miro.medium.com/max/700/1*4tyAmTnvmvCG0xl9PmSLDg.png)
*Image source: https://www.edureka.co/blog/big-data-engineer-skills/*

**3. Real-time Processing Framework:** [Apache Spark](https://spark.apache.org/) — a distributed real-time processing framework which can be easily )integrated with Hadoop leveraging HDFS.

**4. Heavy, In-Depth Database Knowledge — SQL** (e.g. [MySQL](https://www.mysql.com/)) **and NoSQL** (e.g. [HBase](https://hbase.apache.org/), [Cassandra](http://cassandra.apache.org/) and [MongoDB](https://www.mongodb.com/)): Structured Query Language is used to structure, manipulate & manage data stored in databases, while NoSQL databases can store large volumes of structured, semi-structured & unstructured data with quick iteration and agile structure as per application requirements.

**5. Coding Skills:** Python, C/C++, Java, Perl, Golang, or other such languages.

**6. ETL/Data Warehousing Solutions** (e.g. [Talend](https://www.talend.com/), [Informatica](https://www.informatica.com/#fbid=m8rgH3w_yPI)): when managing a huge amount of data from heterogeneous sources you need to apply ETL (Extract Transform Load). Data warehousing helps you aggregate unstructured data from one or more sources to compare and analyze for better business.

**7. Machine Learning:** Although machine learning is technically something assigned to the Data Scientist, having some level of understanding of how to put the data into use using statistical analysis and data modeling is a huge advantage.

**8. Solid Knowledge of Operating Systems:** Apart from extensive knowledge in operating systems such as UNIX, Linux, Solaris or MS Windows can be very helpful as most of the tools will be based on these systems.

**More on this topic:**
[A Beginner’s Guide to Data Engineering](https://medium.com/@rchang/a-beginners-guide-to-data-engineering-part-i-4227c5c457d7) — part I, II and III
[Big Data Engineer Skills: Skills Required To Become A Big Data Engineer](https://www.edureka.co/blog/big-data-engineer-skills/)
[O’Reilly’s Suite of Free Data Engineering E-Books](https://www.oreilly.com/data/free/)


#4. When Does My Team Need a Data Engineer?#

*Tristan Handy* [*@jthandy*](https://twitter.com/jthandy) Founder & CEO @ Fishtown Analytics says that the role of the Data Engineer in a startup data team is changing rapidly which impacts the sequence of Data Engineer hires. Previously we needed Data Engineers first, ‘because data analysts and scientists had nothing to work with if there wasn’t a data platform in place. Today, data analysts and scientists should self-serve and build the first version of their data stack using off-the-shelf tools’.
Handy’ s advice is to hire Data Engineers as you start hitting scale points:

![when does my team need data engineer](https://miro.medium.com/max/622/1*pEjmBoCIpeE-Dvq_er8a_g.jpeg)
*based on https://blog.fishtownanalytics.com/does-my-startup-data-team-need-a-data-engineer-b6f4d68d7da9*


#5. Building Data Team#

![building data team](https://miro.medium.com/max/700/1*O54ez5r7jHgS1YOtvah-zw.png)
*based on https://marketoonist.com/2014/01/big-data.html*

Data-driven teams consist of Data Engineers, Data Scientists, and Data Analysts. While these titles may sound similar, each role focuses on a different aspect of data utilization. It is important to realize how these roles complement each other. Having a Data Scientist doing the job of a Data Engineer and vice versa is a waste of precious resources. Also finding a unicorn — one person who is both skilled at Data Engineering and Data Science seems impossible.

{% twitter 1058015837330784256 %}

*Keith McNulty* [*@dr_keithmcnulty*](http://twitter.com/dr_keithmcnulty) shares his experience and describes in [**6 steps the process of building an analytics team**](https://towardsdatascience.com/how-to-build-an-analytics-team-for-impact-in-an-organization-21bb05925587). These steps advice how the team should operate and be structured, what skills are required, and what types of profiles and skillsets should enter the team.

[*Michael Kaminsky*](https://www.kdnuggets.com/2019/02/analytics-engineer-data-team.html) draws our attention to the fact that the roles and responsibilities of data engineers, analysts, and data scientists are changing.

Data Engineer previously:

*Data engineers: traditionally, this has been a ‘plumbing’ job of moving bytes from point A to point B, typically misnomered simply as ‘ETL’. They were concerned with building robust and scalable infrastructure for ingesting and storing data, but generally did not concern themselves with ‘business logic’— once the data were in the warehouse, it wasn’t their problem anymore.*

Data Engineer today:

*Data engineers: still responsible for data infrastructure and plumbing code, but the team is now generally much smaller than it was in the past. Many companies can get by just using contractors and consultants in the beginning, and they may only need one or two data engineers to ‘fill in the gaps’ of what they can’t purchase from off-the-shelf solutions.*

**More on this topic:**
[How To Build A Big Data Engineering Team](https://www.forbes.com/sites/adrianbridgwater/2017/04/24/how-to-build-a-big-data-engineering-team/#3c1131e95a15)
[How to build an analytics team for impact in an organization](https://towardsdatascience.com/how-to-build-an-analytics-team-for-impact-in-an-organization-21bb05925587)
[Building Data Science Teams: What Do You Need to Know?](https://opendatascience.com/building-data-science-teams-what-do-you-need-to-know/)


#6. Are Data Engineers Expected to also be Data Analysts?#

Lars Kamp from Intermix.io gives us his answer on Quora at [https://qr.ae/TWnwFd](https://qr.ae/TWnwFd).


#7. Data Engineering Job Description#

![data engineering job description](https://miro.medium.com/max/400/1*XJe8bORTujf5_1u4M8C6kQ.png)
*Image source: https://www.equest.com/cartoons/cartoons-2013/thank-you-facebook/*

Whether you are an aspiring Data Engineer or looking for one of them it is always a good habit to use good examples. According to Toptal *‘the actual definition of Data Engineer’s role varies, and often mixes with the Data Scientist role’*. They share their [**Big Data Engineer — Job Description and Ad Template**](https://www.toptal.com/big-data/job-description) you can use to either create a job announcement or to simply review commonly required skills on this position. Another template you will find at [**Glassdoor**](https://www.glassdoor.com/Job-Descriptions/Data-Engineer.htm).

Nevertheless, here’s what Data Engineering **roles typically demand**:

• Build, test, and maintain optimal data pipeline architecture
• Assemble large, complex data sets to meet both functional and non-functional business demands
• Build the infrastructure necessary for optimal extraction, transformation, and loading of data (from a variety of sources leveraging AWS and SQL technologies)
• Identify, design, implement and enhance internal processes
• Automate manual processes
• Optimize data delivery
• Re-design infrastructure for greater scalability
• Build analytics tools that utilize data pipelines to deliver actionable insights
• Work with all stakeholders across departments
• Assist data scientists in building and optimizing products

Another useful tool when screening candidates **for roles within data teams** might be Hackerrank’s [**[Checklist] Screening Data Scientists vs. Analysts vs. Engineers.**](https://blog.hackerrank.com/checklist-screening-data-scientists-vs-analysts-vs-engineers/?utm_medium=content&utm_source=blog&utm_campaign=checklist&utm_content=backendassessment) ‘The nuances between Data Analysts, Data Scientists, Data Engineers may seem minute at first, but each has a distinct role to play in deriving and conveying meaningful insights from data’. The checklist makes a great overview of what to expect from each respective role.

{% twitter 1067818058322984960 %}


#8. Data Engineering Jobs#

![data engineering jobs](https://miro.medium.com/max/400/1*4ZpB7nxMnFjMbQayLyua7Q.png)
*Image source: https://vublsts.wordpress.com/2016/09/28/anyware-privacy-and-location-data-in-the-era-of-machine-learning/*

*While the Harvard Business Review may have declared [‘Data Scientist: The Sexiest Job of the 21st Century,’](https://hbr.org/2012/10/data-scientist-the-sexiest-job-of-the-21st-century) it is the Data Engineering team that allows them to shine. Without the Data Engineering support, the sexy Data Scientist job will quickly devolve into something about as sexy as a street sweeper. — [Bill Schmarzo](https://www.linkedin.com/in/schmarzo/) [@schmarzo](https://twitter.com/schmarzo), CTO Hitachi Vantara.*

Due to the fact that demand for Data Engineers is greater than supply, there is not a problem to find a job in this field. There are hundreds of jobs to scroll down on websites like [Indeed](http://www.indeed.com/q-Data-Engineer-jobs.html), [Linkedin](https://www.linkedin.com/jobs/data-engineer-jobs/), [Glassdoor](http://www.glassdoor.com/Job/data-engineer-jobs-SRCH_KO0,13.htm), [Datajobs](https://datajobs.com/Data-Engineer-Jobs), [Ziprecruiter](http://www.ziprecruiter.com/Jobs/Remote-Data-Engineer), [Angel](https://angel.co/), [Stackoverflow](https://stackoverflow.com/jobs), etc.

But much bigger problem than finding a job in Data Engineering is **finding the candidate** by growing startups. Some of them still need to hire on-site employee, *‘however hiring in big cities like London, New York, Berlin is hard, since for the same limited talent pool there are also fighting corporations having big budgets. Even if you finally manage to hire great techies they will be on the target of other hungry startups crazily looking for devs. And what if you lose your engineers at the peak of product development? Both investors and you would not be happy’.* I encourage you to read the whole article by [*Rob Renner*](https://medium.com/@JavaShopPoland) from [*@JavaShopPoland*](https://twitter.com/JavaShopPoland): [*Why Fintech Startups Should Build Their Development Teams In Poland?*](https://medium.com/javashop/why-fintech-startups-should-move-development-to-poland-b0aa5e9c62ba) together with this one: [*Data engineers are there, can you see them?*](https://towardsdatascience.com/data-engineers-are-there-did-you-see-them-9b6452e92b23) by Guillaume Payen from Moonshots.


#9. Data Engineering Salary#

![data engineering salary](https://miro.medium.com/max/200/1*uTfPMKjwgF3aeUXMWeDeAw.png)
*Image https://pl.pinterest.com/pin/842665780247850785/*

Data Engineers’ salaries depend on variables such as the type of role, skills, experience, and location. According to Glassdoor, the global average salary for a Data Engineer is about [**$116,591 per year**](https://www.glassdoor.com/Salaries/data-engineer-salary-SRCH_KO0,13.htm) while the same index according to Payscale equals [**$91,845 per year**] (https://www.payscale.com/research/US/Job=Data_Engineer/Salary).

According to [Payscale](https://www.payscale.com/research/US/Job=Data_Engineer/Salary) ‘skills in Apache Spark, Hadoop, AWS and ETL are correlated to pay that is above average. Skills that pay less than market rate include Data Analysis and SQL’.

![skills that affect data engineer salary](https://miro.medium.com/max/295/1*ULURARevZ7j-P3kuhIEBTg.jpeg)
*source: payscale.com*

Below a short overview presenting the average Data Engineer’s salary per year in a few locations:

![average data engineer's salary per year](https://miro.medium.com/max/587/1*ZjJ5xY58PJvrLBAAcBxCBQ.jpeg)
*source: based on payscale.com and neuvoo.ch*


#10. Data Engineering Interview Question#

If you are looking for a job that is related to Data Engineering, you need to prepare for the interview questions you might face during the meeting or a call.

There are a few self-study pages recommended by us you should definitely visit to prepare for this conversation.

1. [Data Engineer Interview Questions](https://www.educba.com/data-engineer-interview-questions/) — the list of top 10 Data Engineer Interview Questions and Answers Updated for 2019

2. [Things you should know when traveling via the Big Data Engineering hype-train ](https://www.kdnuggets.com/2018/10/big-data-engineering-hype-train.html)—really interesting article by *Wojtek Pituła* [*@Krever01*](https://twitter.com/Krever01)

3. [The Interview Study Guide For Data Engineers](https://www.coriers.com/the-interview-study-guide-for-data-engineers/) — with a practical Downloadable Data Engineering Interview Checklist

4. [Facebook Data Engineer Interview Questions](https://www.glassdoor.com/Interview/Facebook-Data-Engineer-Interview-Questions-EI_IE40772.0,8_KO9,22.htm) & [Amazon Data Engineer Interview Questions](https://www.glassdoor.ie/Interview/Amazon-Data-Engineer-Interview-Questions-EI_IE6036.0,6_KO7,20.htm) — covering real-life examples of Facebook and Amazon Data Engineer Interview Questions

5. [30+ Best Data Engineer Interview Questions To Get Hired With](https://www.algrim.co/posts/154-data-engineer-interview-questions) — another set of 33 Q&A

6. 1001 Data Engineering Interview Questions by Andreas Kretz also available on [Github](https://github.com/andkret/Cookbook) in PDF [from page 111].

{% youtube WbqRH2r3N40 %}


#11. Data Engineering Certification#

![data engineering certification](https://miro.medium.com/max/700/1*_hEDASII5u7vgqJ9NH4mYA.jpeg)
*image source: https://unsplash.com/photos/TamMbr4okv4*

If you are looking for the most demanded certifications in Data Engineering look at this awesome list by [Thor Olavsrud](https://www.cio.com/author/Thor-Olavsrud/) [@ThorOlavsrud](https://twitter.com/ThorOlavsrud). He provides a comprehensive overview of TOP 14 certifications including price, hosting organization, and how to prepare.

And here are the **Top 14 Data Engineer and Data Architect Certifications:**

1.[*Amazon Web Services (AWS) Certified Big Data — Specialty*](https://aws.amazon.com/certification/certified-big-data-specialty/)

2.[*Cloudera Certified Associate (CCA) Spark and Hadoop Developer*](https://www.cloudera.com/about/training/certification/cca-spark.html)

3.[*Cloudera Certified Professional (CCP): Data Engineer*](https://www.cloudera.com/about/training/certification/ccp-data-engineer.html)

4.[*Google Professional Data Engineer*](https://cloud.google.com/certification/data-engineer)

5.[*HDP Apache Spark Developer*](https://www.cloudera.com/about/training/certification/hdp-cert-spark-dev.html)

6.[*HDP Certified Developer Big Data Hadoop*](https://www.cloudera.com/about/training/certification/hdp-cert-dev.html)

7.[*Hortonworks Certified Associate (HCA)*](https://www.cloudera.com/about/training/certification.html)

8.[*IBM Certified Data Architect — Big Data*](https://www.ibm.com/certify/cert?id=50001701)

9.[*IBM Certified Data Engineer — Big Data*](https://www.ibm.com/certify/cert?id=50001501)

10.[*MapR Certified Hadoop Developer 1.0*](https://mapr.com/training/certification/)

11.[*MapR Certified Spark Developer 2.1*](https://mapr.com/training/certification/)

12.[*Oracle Business Intelligence Foundation Suite 11 Certified Implementation Specialist*](https://education.oracle.com/oracle-business-intelligence-foundation-suite-11g-certified-implementation-specialist/trackp_166)

13.[*SAS Certified Big Data Professional*](https://www.sas.com/en_us/training/academy-data-science/big-data-certification.html)

14.[*SAS Certified Data Scientist Using SAS 9*](https://www.sas.com/en_us/certification/credentials/advanced-analytics/data-scientist.html)

source: [*Top 14 data engineer and data architect certifications*](https://www.cio.com/article/3395879/top-14-data-engineer-and-data-architect-certifications.html)

**More on this topic:**
[Certificates and Certification in Analytics, Big Data, Data Science, Machine Learning](https://www.kdnuggets.com/education/analytics-data-mining-certificates.html)



#12. Data Engineering [Online] Courses#

![data engineering online courses](https://miro.medium.com/max/400/1*R_IdZY_YsCnWGDBDDN5vEA.png)
*Image source: canva.com*

There are many online courses offering significant training in this field:
• [Udemy](https://www.udemy.com/courses/search/?q=data%20engineering&src=ukw)
• [Coursera](https://www.coursera.org/)
• [Dataquest](https://www.dataquest.io/)
• [EdX](https://www.edx.org/course/introduction-apache-spark-uc-berkeleyx-cs105x)
• [Memrise](https://www.memrise.com/course/584043/r-for-data-analysis)
• [DataCamp](https://www.datacamp.com/)
• [Galvanize](http://www.galvanize.com/)

Some sites, such as [DataCamp](https://www.datacamp.com/), [Memrise](https://www.memrise.com/course/584043/r-for-data-analysis) and [EdX](https://www.edx.org/course/introduction-apache-spark-uc-berkeleyx-cs105x) are heavily focused on data science and engineering, while others, such as [Udemy](https://www.udemy.com/courses/search/?q=data%20engineering&src=ukw) and [Galvanize](http://www.galvanize.com/), have a wider scope. Your choice of course provider will depend on your needs, level of knowledge and your wallet size.

TIP: Although a course can help you expand your knowledge it may only offer you a certificate or diploma rather than a certification. Therefore do not treat a course as a replacement for actual certification or accredited diploma issuance.

**More on this topic:**
[Learn Data Engineering: My Favorite Free Resources](https://hackernoon.com/learn-data-engineering-my-favorite-free-resources-52a29ab999b)
[Want to Become a Data Engineer? Here’s a Comprehensive List of Resources to get Started](https://www.analyticsvidhya.com/blog/2018/11/data-engineer-comprehensive-list-resources-get-started/)


#13. Data Engineering Podcasts#

Constant innovations can overwhelm anyone, but in tech, if you’re not learning, you’re becoming obsolete. Therefore, to stay up to date with what is going on in Data world try listening to these 5 podcasts’ resources:

1.[Data Skeptic](https://www.stitcher.com/podcast/data-skeptic-podcast/the-data-skeptic-podcast)
2.[The O’Reilly Data Show](https://www.stitcher.com/podcast/oreilly-media-2/the-oreilly-data-show-podcast?refid=stpr)
3.[Digital Analytics Power Hour](https://www.stitcher.com/podcast/httpanalyticshourlibsyncom/the-digital-analytics-power-hour)
4.[Data Engineering Podcast](https://www.dataengineeringpodcast.com/)
5.[Data Stories](https://www.stitcher.com/podcast/data-stories-podcast/data-stories)

However, if you prefer watching & listening than only listening go to [*Webcasts / Webinars on AI, Analytics, Big Data, Data Science, & Machine Learning*](https://www.kdnuggets.com/webcasts/index.html) by KdNuggets — an up-to-date list of webinars or watch [*Thomas Henson’s YouTube channel*](https://www.youtube.com/user/PirateDevelopment).


#14. Data Engineering Case Studies#

As for this point, there is a comprehensive case study collection created by [*Andreas Kretz in his Data Engineering CookBook*](https://github.com/andkret/Cookbook). You will find here a great number of examples of companies like Twitter, Netflix, Amazon, Uber, Airbnb, and many other prominent players. Some of them are also available on Youtube. Download the [*PDF*](https://github.com/andkret/Cookbook) and follow the list of contents to find the required resources.

{% twitter 1153447958895091712 %}


#15. Data Engineering Conferences 2019#

![data engineering conferences](https://miro.medium.com/max/584/1*l-v1dxIkX3YjlZ3UJTWS7w.jpeg)
*Image source: https://twitter.com/Khanoisseur/status/556881894366003200?s=20*


[KDnuggets](https://www.kdnuggets.com/), one of the biggest websites in the Big Data and data science space provides us with a complete and updated list of [*Meetings and Conferences on AI, Analytics, Big Data, Data Mining, Data Science, & Machine Learning*](https://www.kdnuggets.com/meetings/index.html#.XTCk5DZqE-A.twitter) broken down by location in America, Europe, and Asia.

If it happens you find even more widespread list of conferences just put it in a comment.

#16. Top 10 Data Engineering Mistakes#

{% youtube Hyhwem1Gyjo %}

Other suggestions?


#17. Recommended Resources#

[*Top Active Blogs on AI, Analytics, Big Data, Data Science, Machine Learning — updated*](https://www.kdnuggets.com/2019/01/active-blogs-ai-analytics-data-science.html)

[*The Ultimate Guide to AI, Data Science & Machine Learning, Articles, Cheatsheets and Tutorials ALL in one place*](https://www.linkedin.com/pulse/all-cheatsheets-one-place-vipul-patel/)

[*Data-Science — Cheat-Sheet/Data Engineering*](https://github.com/abhat222/Data-Science--Cheat-Sheet/tree/master/Data%20Engineering)

[*Data Science 101*](https://101.datascience.community/)


#Data Is Just The Beginning…#

![data is just the beginning](https://miro.medium.com/max/432/1*juK_NS4KRKOAUj5yhqOlXA.jpeg)
*Image source: https://everythingfunny.org/funny-cartoons/customer-who-bought-this*

Originally posted on Medium: [Towards Data Science: Data Engineering Complete Reference Guide](https://towardsdatascience.com/data-engineering-complete-reference-guide-from-a-z-2019-852c308b15ed)