Demystifying Metadata Management — Part 1
Tags: bigdataplatform, datawarehousesolutio, datalake, metadatamanagement
Published: 2023-11-03 14:49:22

---
title: Demystifying Metadata Management — Part 1
published: true
date: 2023-01-17 10:27:02 UTC
tags: bigdataplatform,datawarehousesolutio,datalake,metadatamanagement
canonical_url: 
---

### Demystifying Metadata Management — Part 1

Metadata Management provides a base for an organization’s Data Platform Architecture. Let’s understand each component and its role in metadata Management.

![](https://cdn-images-1.medium.com/max/1024/1*M4hqmXco3kRYXUX7ewfZsg.png)
_Image courtesy infopulse.com_

**Data:**

Data is a collection of raw and unorganized facts that can be used in calculating, reasoning or planning. Without proper processing and organizing, it is useless. That’s where metadata comes into play.

_Good read on Data:_ [_Blog by Dataedo_](https://dataedo.com/blog/data-vs-metadata)

![](https://cdn-images-1.medium.com/max/1024/1*V7AgJk1qJYTyNgLU0aS44Q.png)
_Image Courtesy: Dataedo by @piotr kononow_

### **MetaData:**

Metadata is simply data about data. It means it is a description and context of the data. It helps to organize, find and understand data, through information such as format, origin, creation date, modification date, etc.

Data stores information, but if you don’t know how to interpret it, you don’t have access to this information. Metadata enables you to understand data and extract the information.

> Metadata, you see, is really a love note — it might be to yourself, but in fact it’s a love note to the person after you, or the machine after you, where you’ve saved someone that amount of time to find something by telling them what this thing is.

> _Cit._ [_Jason Scott’s Weblog_](http://ascii.textfiles.com/archives/3181)

![](https://cdn-images-1.medium.com/max/1024/1*AzhWcFKZSTy_OGHKhulb9g.png)
_Image Courtesy: ontotext.com_

_Good read on metadata:_ [_Blog by Dataedo_](https://dataedo.com/blog/data-vs-metadata)

### **Data Democratization:**

Empowering employees and stakeholders of an organization with the right set of tools that enables them to make informed decisions.

Data democratization is the _ongoing process_ of enabling everybody in an organization, irrespective of their technical knowledge — how, to _work with data_ _comfortably_, to feel _confident talking about it_, and as a result, _make data-informed decisions_ and _build customer experiences powered by data._

> **Data Democraatization have answers to questions like:**

> “Experts in my company are too busy to help me”.

> “I do not have access to data”

> “I can not trust the data”.

Data democratizaton is an ongoing process and need cultural shift because it depends on ongoing process called _Data Literacy._

![](https://cdn-images-1.medium.com/max/1024/1*5lXSSIK9g_x1D8mS7n8bYw.png)
_Image Courtesy: Arpit Choudhury from his medium blog_

_Good read on Data Democratization_: [_Blog by Towards datascience_](https://towardsdatascience.com/what-the-heck-is-data-democratization-39b86eb27aa6)

### **Data Literacy:**

The ability to _read, analyze, work_ and _communicate_ with data — known as data literacy — is now so critical to companies that it has been hailed as the second language of business by [Gartner](https://www.gartner.com/smarterwithgartner/a-data-and-analytics-leaders-guide-to-data-literacy#:~:text=Gartner%20defines%20data%20literacy%20as,case%2C%20application%20and%20resulting%20value.). The global pandemic highlighted its importance, with many companies starting to rely on data to detect new patterns, respond to changing customer behavior and make first-of-a-kind decisions in a new environment of many unknown factors.

> Poor data literacy is ranked as the second-biggest internal roadblock to the success of the CDO’s office, according to the Gartner Annual Chief Data Officer Survey.

In upcoming years, data literacy will become essential in driving business value, demonstrated by its formal inclusion in over 80% of data and analytics strategies and change management programs.

One common misconception about Data Democratization and Literacy is that now everyone in the compaby will know everything related to the data and get you details about data in no time and there will be no need for _Subject Matter Expert_ or _Data Architect_. This is not true.

Data Literacy and Democratiaztion provides way to be independent and able to complete tasks and take company to the right direction and have no place for presumption.

_Good read on Data Literacy:_ [_Blog by thedataliteracyproject_](https://thedataliteracyproject.org/files/downloads/The%20Seven%20Principles%20of%20Data%20Literacy%20by%20Qlik%20and%20Accenture.pdf)

![](https://cdn-images-1.medium.com/max/1024/1*nWPDxu1IOTQ83S6fHxBepg.png)
_Image Courtesy: Dataedo by @piotr kononow_

### Data Architect & Data Engineer:

[The data architect and data engineer titles](https://www.striim.com/blog/data-architect-vs-data-engineer-an-overview-of-two-in-demand-roles/#:~:text=The%20data%20architect%20and%20data%20engineer%20titles%20are%20closely%20related,responsible%20for%20creating%20that%20vision.) are closely related and, as such, frequently confused. The difference in both roles lies in their primary responsibilities.

Data architects design the vision and blueprint of the organization’s data framework, while the data engineer is responsible for creating that vision.

Data architects provide technical expertise and guide data teams on bringing business requirements to life; data engineers ensure data is readily available, secure, and accessible to stakeholders (data scientists, data analysts) when they need it.

Data architects have substantial experience in data modeling, data integration, and data design and are often experienced in other data roles; data engineers have a strong foundation in programming with software engineering experience.

The data architect and the data engineer work together to build the organization’s data system.

_Good read on Data Architect vs Data Engineer:_ [_Blog by rsTask_](https://rstak.medium.com/data-science-vs-data-engineering-vs-data-architecture-3a05785f7f48)

![](https://cdn-images-1.medium.com/max/1024/1*4EewBA3kzi8hnQOvQSMd9w.png)
_Image Courtesy: Arun Elangovan_

### Data Steward & Data Analyst & Data Scientist:

- **_Data Analyst_** gather data from various databases and warehouses, filter and clean it. **_Data Scientist_** perform ad-hoc data mining and gather large sets of structured and unstructured data from several sources.
- **_Data Analyst_** write complex SQL queries and scripts to collect, store, manipulate, and retrieve data from RDBMS such as MS SQL Server, Oracle DB, and MySQL. **_Data Scientist_** use various statistical methods, data visualization techniques to design and evaluate advanced statistical models from vast volumes of data.
- **_Data Analyst_** create different reports with the help of charts and graphs using Excel and BI tools. **_Data Scientist_** Build AI models using various algorithms and in-built libraries.
- **_Data Analyst s_** pot trends and patterns from complex datasets. **_Data Scientist_** Automate tedious tasks and generate insights using machine learning models.

At high level, Data Steward handles day to day operations on policeis created by either Data Architect.

![](https://cdn-images-1.medium.com/max/866/0*LNH7cHTprLTsjN4s.png)
_Data Engineers are the Bridge by Jennifer Shalamanov_

The data steward is the “go-to” guy for everyone working with data within the company. Typical data steward roles and responsibilities can be grouped as:

- Operational Oversight — a data steward oversees the lifecycle of a data set. They are responsible for defining and implementing rules and regulations for the day-to-day operational and administrative management of data and systems.
- [Data Quality](https://www.simplilearn.com/data-quality-article) — data steward responsibilities include establishing data quality metrics and requirements, like setting acceptable values, ranges, and parameters for every data element.
- Privacy, Security, and Risk Management — data protection is a key aspect of data steward responsibilities. A steward must establish regulations and conventions that govern data proliferation to ensure that data privacy controls are exercised in all processes.
- Policies and Procedures — data stewards, also establish policies and procedures for data access, including authorization criteria based on any individual and/or the role.

_Good read on Data Steward vs Data Analyst:_ [_Blog by Simplilearn_](https://www.simplilearn.com/what-is-data-stewardship-article#:~:text=Data%20Analyst%3A%20How%20the%20Two,patterns%2C%20and%20predict%20future%20results.)

### **Data Warehouse & Data Lake & Data Mart:**

**Data warehouse (DW)** is a system for aggregating data from connected databases — and then transforming and storing it in an analytics-ready state. The main benefits of a data warehouse are effective data consolidation, fast pre-processing, and easy self-access for business users. _The key constraint of using a data warehouse solution is the need to pre-transform all data using standard schemas. This increases the usage costs and reduces scalability potential._

**Data warehouse solutions:**

- Azure Synapse Analytics
- Amazon Redshift
- Google BigQuery
- Snowflake

![](https://cdn-images-1.medium.com/max/772/1*A7BOcfgiLJeqqfAUXa4q1A.png)
_Image Courtesy: Dataedo by @piotr kononow_

**Data lake** is a centralized cloud-based repository for storing raw (unprocessed, non-cataloged, or pre-cleansed) data from various systems. Unlike DWHs, data lake technology allows storing both structured and unstructured data of any size (as object blobs or files). Cloud data lakes are also more scalable and support more querying methods for data retrieval and analysis — a factor data scientists well appreciate.

- **Data lake solutions:**
- Azure Data Lake
- Amazon S3
- Apache Hadoop

**Data Mart** is more focused subset of data present in Data Warehouse. It generally concerned with a single team of department like finance, marketing, or sales. It is smaller, more focused, and may contain summaries of data that best serve its community of users. A data mart might be a portion of a data warehouse, too.

Data Mart has few benefits over giving access to fuill warehouse to all the departments:

- Cost-efficiency
- Simplified data access
- Quicker access to insights
- Simpler data maintenance
- Easier and faster implementation

_Good read on Data Warehouse vs DataLake:_ [_Blog by AWS_](https://aws.amazon.com/data-warehouse/)

### Conclusion:

This is the first part of series on metadata management. This part will help in building conceptual blocks of metadata management.

Please stay tuned for more parts of the series where we will discuss metadata management in detail and will also take one example of an organization to create metadata management for an example organization.

Please comment if you guys want me to focus on metadata management of any specific industry like E-Commerce, Healthcare, or Offline retail.

**Keep Learning:** Please refer here for part of the [_Demystifying Metadata Management — Part 2_](https://medium.com/@arglee/demystifying-metadata-management-part-2-378434352bdb)_._