What is Data?
Tags: database, data, sre, programming
Published: 2023-05-09 22:47:48

There are three categories of data: structured, semi-structured, and unstructured. Each type of data has its own characteristics and use cases, understanding the differences between them is crucial for effective data management and analysis.

**Structured data** is organized and easily searchable. It is typically stored in _relational databases_, and its format is well-defined with pre-determined columns, data types, and relationships. 

Examples include data from _enterprise resource planning (ERP) systems_, _customer relationship management (CRM) databases_, and _financial records_. Structured data can be easily queried and analyzed using **SQL** and other database tools. An example of Structured Data:

```
import sqlite3

# Connect to the database
conn = sqlite3.connect('example.db')

# Create a table with structured data
conn.execute('''CREATE TABLE employees
             (id INT PRIMARY KEY NOT NULL,
             name TEXT NOT NULL,
             age INT NOT NULL);''')

# Insert data into the table
conn.execute("INSERT INTO employees (id, name, age) VALUES (1, 'John Doe', 25)")
conn.execute("INSERT INTO employees (id, name, age) VALUES (2, 'Jane Smith', 30)")

# Query the data from the table
cursor = conn.execute("SELECT * FROM employees")
for row in cursor:
    print("ID = ", row[0])
    print("Name = ", row[1])
    print("Age = ", row[2])

# Close the database connection
conn.close()
```
>  A structured database table with pre-defined columns for id, name, and age. We insert data into the table and query it using SQL.

---

**Semi-structured data** falls somewhere between structured and unstructured data. It has a defined structure, but it's not as rigid as structured data. 

Semi-structured data _often includes metadata and tags_ that provide additional context. Examples include **XML** and **JSON** files, which are commonly used to exchange data between web applications.

```
import json

# Define a JSON object with semi-structured data
employee = {
  "id": 1,
  "name": "John Doe",
  "age": 25,
  "department": {
    "name": "Engineering",
    "manager": "Jane Smith"
  }
}

# Convert the JSON object to a string
employee_json = json.dumps(employee)

# Print the JSON string
print(employee_json)

# Convert the JSON string back to a Python object
employee_dict = json.loads(employee_json)

# Access the data in the Python object
print(employee_dict['id'])
print(employee_dict['name'])
print(employee_dict['age'])
print(employee_dict['department']['name'])
print(employee_dict['department']['manager'])
```

> we define a JSON object with semi-structured data that includes a nested department object. We convert the object to a JSON string and back to a Python object, accessing the data using dictionary keys.

---

**Unstructured data lacks any predefined structure**. It's the most challenging type of data to work with because _it includes text, images, and multimedia files_ that don't fit neatly into a database schema.

Examples include emails, social media posts, images, and videos. Unstructured data can be challenging to analyze using traditional data analysis tools, but advancements in natural language processing [NLP](https://www.ibm.com/topics/natural-language-processing) and machine learning algorithms are making it easier to derive insights from unstructured data.

```
import pytesseract
from PIL import Image

# Open an image file with unstructured data
image = Image.open('example.png')

# Use Tesseract OCR to extract text from the image
text = pytesseract.image_to_string(image)

# Print the extracted text
print(text)
```
> we open an image file with unstructured data and use Tesseract OCR to extract text from the image. The extracted text doesn't have a pre-defined structure and is challenging to analyze without advanced NLP techniques.

---

Each datatype have their unique characteristics and uses. Programmers and organizations can better manage and analyze their data by understanding the usecases, costs and benefits of each datatype, leading to more informed decision-making and better business outcomes. 

A great resource to explore for learning about working with within the Azure ecosystem is the [Microsoft Certificate: Azure Data fundamentals](https://learn.microsoft.com/en-us/certifications/azure-data-fundamentals/). It looks like this course is even [freely available for students](https://techcommunity.microsoft.com/t5/educator-developer-blog/how-to-get-a-microsoft-certification-for-free-as-a-student/ba-p/3584897).

Something I happened across today to add context to how the industry is thinking about data and datatypes in [the ongoing conversation around AI](https://a16z.simplecast.com/episodes/from-promise-to-reality-inside-a16zs-data-and-ai-forum)