Data Engineering skills
Tags: bigdata, bi, mlops, data
Published: 2021-03-14 15:14:40

---
title: Data Engineering skills
published: true
description: To solve any big data problem you need proper infrastructure, monitoring, logging and solve other operational challenges
tags: bigdata, bi, mlops, data
cover_image: https://dev-to-uploads.s3.amazonaws.com/uploads/articles/zasabhmijefdr3dwgfpq.JPG
---


My friend asked me an interesting question about what skills are worth learning for Data Management specialists and how to build a [grow roadmap](https://github.com/datastacktv/data-engineer-roadmap). 

In fact, the question made me think because I haven't had a clear picture in my head. It's just my thoughts on the topic and for the most part, I'm just **speculating** about the current state and the future of Data Management. 

---

## Prerequisites

In the beginning, as in any other area, there are basic things that any Software Engineer should know.

In short, I assume that the person who came to the Big Data already knows some programming language and has an idea about such basic things as CS algorithms, SQL, VCS, SDLC, Networking fundamentals, Linux basics, [CI/CD](https://luminousmen.com/post/continuous-Integration-continuous-delivery), etc. One way or another the common practices of software engineering do not go anywhere — this is the basis that I think is needed in any field. If you lack them then you better learn them, but perhaps you can argue with me here.

## Big Data Basics

This is another layer of knowledge that I think is the basic one. Here I could highlight programming languages, for example, because in the Big Data world there are predominant languages without which it's just hard (or impossible) to do something. These are of course JVM languages such as **Java**, **Scala** may soon be joined by Kotlin, and, of course, **Python** as the second best language for everything and the de-facto language for Data Science. 

Aside from languages, there are some terms and fundamental concepts behind the Big Data that I think should be learned such as horizontal scaling, distributed systems, [asynchronous communication](https://luminousmen.com/?tag=asynchrony), CAP theorem, eventual consistency, consensus problems, availability, durability, reliability, resiliency, fault tolerance, disaster recovery, etc. There is no need for a deep understanding of them but if you want to succeed in the field you better Google it at least. 

Now let's talk about how the field is developing, to be more precise, the challenges it faces.

## Data Challenges

[Data Challenges in Big Data](https://luminousmen.com/post/data-challenges-in-big-data)

## Management Challenges

[Management Challenges in Big Data](https://luminousmen.com/post/management-challenges-in-big-data)

## Analytical Challenges

[Analytical Challenges in Big Data](https://luminousmen.com/post/analytical-challenges-in-big-data)

## Operational Challenges in Big Data

[Operational Challenges in Big Data](https://luminousmen.com/post/operational-challenges-in-big-data)

## Conclusion

It turned out a lot and it seems that I did not say anything. 

Too many products are available. Most of them claim to solve all data problems your company encounter. But it is not true.

I do not consider myself an expert in everything and only speculate on technology here. But as you can see even from my article, many skills overlap in several areas of Big Data and do not end there. Having them, you can not be afraid that you will not find a job.

*Do not chase trends — build skills that stay relevant. And the most relevant skills are probably soft skills.*

The steep learning curve associated with a lot of data engineering activities becoming a cliff. Developers' hand-coding projects need deep knowledge of many aspects of the organization and a lot of tools and existing solutions. 

In data management, we are still on the Wild West, especially in ML...

---

**Thank you for reading!**

Any questions? Leave your comment below to start fantastic discussions!

Check out [my blog](https://luminousmen.com/) or come to say hi 👋 on [Twitter](https://twitter.com/luminousmen) or subscribe to [my telegram channel](https://t.me/iamluminousmen).Plan your best!