Let Fabric teach you how to code with Data Wrangler
Published: 2023-08-24 15:00:11

---
title: Let Fabric teach you how to code with Data Wrangler
published: true
date: 2023-08-24 07:34:53 UTC
tags: 
canonical_url: https://dataroots.io/research/contributions/let-fabric-teach-you-how-to-code-with-data-wrangler
---

I'm going to be honest with you. I'm bad at writing Pandas data transformation code. Throughout the years I mostly focussed on the Spark APIs in Scala and PySpark, SQL, dbt, and some others, but I find the Pandas APIs usually just confusing and hard to read. I don't like the black box magic and lock-in of low-code solutions either. Did you know that Microsoft Fabric has the perfect middle ground for this? It's called Data Wrangler. Let's dive in!

Actually, Data Wrangler is not a new thing. It was built as an extension for Visual Studio Code. It assists data engineers to wrangle their data with an intuitive interface that helps you write Pandas code. Microsoft nicely integrated this into the Fabric experience to make it even easier to use.

To get started with Data Wangler, create a new notebook in your Fabric Workspace. A notebook needs a Fabric Lakehouse and in our tutorial, we're going to need some data to wrangle. I uploaded [this CSV file](https://debruyn.dev/2023/let-fabric-teach-you-how-to-code-with-data-wrangler/?ref=dataroots.ghost.io#:~:text=wrangle.%20I%20uploaded-,this%20CSV,-file%20with%20the) with the most popular movies & series on IMDB to my Lakehouse and linked the Lakehouse to my notebook. To get started with wrangling data, we need to load our CSV into a Pandas DataFrame. The syntax to read a CSV file with Pandas is pretty straightforward, but you can save yourself the hassle by right-clicking on the CSV file.

![](https://dataroots.ghost.io/content/images/2023/08/generate-pandas-df.png)
_Generate the code to load the CSV file_

This puts the code to load the CSV file in the first cell so all you have to do is click the :play\_button: button to execute the code. You can see the result right below.

![](https://dataroots.ghost.io/content/images/2023/08/generated-pandas-df.png)
_Load the DataFrame_

This has now loaded the DataFrame into the current session and we can now start wrangling! To get to the Data Wangler, click the _Data_ button and select the DataFrame (`df` in my example) you want to wrangle.

![](https://dataroots.ghost.io/content/images/2023/08/launch-data-wrangler.png)
_Select your DataFrame to open the Data Wrangler_

Instantly, you get thrown into a clarifying overview of your data. You get statistical facts, nice histograms, and lots more without having to write a single line of code. Amazing!

![](https://dataroots.ghost.io/content/images/2023/08/data-wrangler-overview.png)
_Data Wrangler overview_

But my data is not clean. So how do we start? At the left bottom corner, you see a list of operations currently applied to your DataFrame. That list is empty now, but we're going to add some basic data cleansing operations to it. Let's start with a basic one: dropping all rows with missing values and dropping all duplicate rows. All we have to do is browse through the list of available operations in the menu on the left and pick the operation we want to apply. For every operation, there are some parameters you can set. In our case, we had to identify the columns which should never be empty or the columns used to detect duplicates. Easy peasy!

![](https://dataroots.ghost.io/content/images/2023/08/operation-drop-duplicate-rows.png)
_Drop duplicate rows_

The Fabric Data Wangler instantly calculates and shows us the results. Where is the code now, you ask? Well, it's right there at the bottom of your screen while you're configuring an operation. If you now want to put that code into your notebook, the button _Add code to notebook_ allows you to do so.

![](https://dataroots.ghost.io/content/images/2023/08/add-code-to-notebook.png)
_Add the code to your notebook_

This takes you back to the notebook, and well ... you guessed it, the code is there!

For me it generated this cell:

```
def clean_data(df):
    # Drop rows with missing data in columns: 'MOVIES', 'YEAR'
    df = df.dropna(subset=['MOVIES', 'YEAR'])
    # Drop duplicate rows in columns: 'MOVIES', 'YEAR'
    df = df.drop_duplicates(subset=['MOVIES', 'YEAR'])
    # Change column type to string for column: 'YEAR'
    df = df.astype({'YEAR': 'string'})
    return df

df_clean = clean_data(df.copy())
df_clean.head()

```

> What stands out to me is that even generated comments to explain what the code does. This is a great way to learn Pandas code!

I couldn't get enough of this productive and fun way to wrangle my data and learn Pandas code. Next, I wanted to fix the `YEAR` column. For this, I decided to write the code myself, so I just added a new cell and a few lines to remove the characters I didn't want to have in that column:

```
df_clean["YEAR"] = df_clean["YEAR"].str.replace("(", "")
df_clean["YEAR"] = df_clean["YEAR"].str.replace(")", "")
df_clean["YEAR"] = df_clean["YEAR"].str.replace(" ", "")
display(df_clean)

```

Wrangling more data? Just execute all cells and you can see the new `df_clean` pop-up in the same menu as before to open the Data Wrangler.

![](https://dataroots.ghost.io/content/images/2023/08/launch-data-wrangler-2.png)
_Going back to the Data Wangler_

My `YEAR` column is far from finished. Sometimes it contains a start and an end, and sometimes only a single year. Let's split that. The Data Wrangler in Fabric has an operation to do exactly that. While we're at it, let's do the same with the `GENRE` column as well.

![](https://dataroots.ghost.io/content/images/2023/08/split-column-preview-year.png)
_Split the year column_

What's really nice about this, is that you instantly get to see a preview of your operation so that you can fine-tune the parameters to get the result you want. I had no idea how many genres a movie or series could have, but what do you know, it's 3!

![](https://dataroots.ghost.io/content/images/2023/08/split-column-preview-genre.png)
_Split the genre column_

I finished my wrangling by renaming some columns and sorting by the rating. This time I didn't have to go through the operations menu, I could just right-click on the column and click _Rename_ or _Sort_.

Now, let's see how all of this translates to code! I clicked the _Add code to notebook_ button again and it brought me back to my notebook.

![](https://dataroots.ghost.io/content/images/2023/08/notebook-with-code.png)
_The completed notebook_

Once again it nicely documented my code as well:

```
# Split text using string '–' in column: 'YEAR'
df_split = df_clean['YEAR'].str.split(pat='–', expand=True).add_prefix('YEAR_')
df_clean = pd.concat([df_clean.iloc[:, :2], df_split, df_clean.iloc[:, 2:]], axis=1)
df_clean = df_clean.drop(columns=['YEAR'])
# Rename column 'YEAR_0' to 'year'
df_clean = df_clean.rename(columns={'YEAR_0': 'year'})
# Rename column 'MOVIES' to 'title'
df_clean = df_clean.rename(columns={'MOVIES': 'title'})
# Rename column 'YEAR_1' to 'year_end'
df_clean = df_clean.rename(columns={'YEAR_1': 'year_end'})
# Split text using string ',' in column: 'GENRE'
df_split = df_clean['GENRE'].str.split(pat=',', expand=True).add_prefix('GENRE_')
df_clean = pd.concat([df_clean.iloc[:, :4], df_split, df_clean.iloc[:, 4:]], axis=1)
df_clean = df_clean.drop(columns=['GENRE'])
# Rename column 'GENRE_0' to 'genre_0'
df_clean = df_clean.rename(columns={'GENRE_0': 'genre_0'})
# Rename column 'GENRE_1' to 'genre_1'
df_clean = df_clean.rename(columns={'GENRE_1': 'genre_1'})
# Rename column 'GENRE_2' to 'genre_2'
df_clean = df_clean.rename(columns={'GENRE_2': 'genre_2'})
# Remove leading and trailing whitespace in columns: 'genre_0', 'genre_1', 'genre_2'
df_clean['genre_0'] = df_clean['genre_0'].str.strip()
df_clean['genre_1'] = df_clean['genre_1'].str.strip()
df_clean['genre_2'] = df_clean['genre_2'].str.strip()
# Rename column 'RATING' to 'rating'
df_clean = df_clean.rename(columns={'RATING': 'rating'})
# Rename column 'VOTES' to 'vote_count'
df_clean = df_clean.rename(columns={'VOTES': 'vote_count'})
# Sort by column: 'rating' (descending)
df_clean = df_clean.sort_values(['rating'], ascending=[False])

```

I'm really impressed with the Data Wrangler in Fabric. It's a great way to learn Pandas code and it's a great way to quickly understand and clean your data. I'm definitely going to use this more often!

# You might also like

[

Connect to Fabric Lakehouses & Warehouses from Python code - Sam Debruyn

In this post, I will show you how to connect to your Microsoft Fabric Lakehouses and Warehouses from Python. Packages & dependencies To connect to Fabric, we’ll use the Microsoft ODBC Driver. This driver is available for Windows, Linux, and macOS. Click on your operating system to download and in…

 ![](data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAACAAAAAgCAYAAABzenr0AAAABGdBTUEAALGPC/xhBQAAACBjSFJNAAB6JgAAgIQAAPoAAACA6AAAdTAAAOpgAAA6mAAAF3CculE8AAAABmJLR0QA/wD/AP+gvaeTAAAACXBIWXMAAAsSAAALEgHS3X78AAAAB3RJTUUH5QMTDQUuJijI6QAAAAFvck5UAc+id5oAAANWSURBVFjD7dZZiJVlGAfw33GGaWJomMzItEXbbHGGZoyiRW0RLxLypgutbiuIFiJbQCKKiGihneqiBW+CoGIIuqgkEJexxZRogbLQjJrKaLFi1Jm6eN7Pec93vnOcAaOb+cOB873b83+2//syhf8ZtUNyyuCDreeX3/UfEqg3PhPz8AWGJ0KmOYGyV1Ve1K+Zg2dwCbZjPdbiA+zCaBWZagITCWn9mjPwLBaXVu7HTryPd7ERX2OkOYHxg9txBcaSF9+n/2UM4Hmc05q1f0RaPsLd+LgwUuV1B25NCzsS66HkxRB2YB8WJs/PKnm9Bz0lAjVRI8uwqSBQqzB+OFbj9mQ8xxi+SxH5FCtxSjY/gocxiPNxGRZgFqZl617DCuyvlYx3437cgDaTw5+4F4+n6BSRPBFLcB9mpPHPcCmGc1Yz8ARuLBnfKfLfCr9gFR7NjMNefIkXsC0bPw5zGa+BWcn4laWD38MtIq8XpZCei+Pl9cOTeO7AV96yEeG92Jr2F5Huw1B7YvI0Li8ZfxM3iYKDz/EiZqfxVdnawyqN12OLqKEi6gPSx9kZM2nRK7guM15gNKXkDVHpBRag8yBp+iSlqkAvetqTZ8M4IU3sEu33Q4uQfiO6YV4aPQ3HNBCuL/A2hQAFTsKcdnyLrzIC3ThKyGmzkP4s9L4gMBMnY0cTFe0VaZ6djfVg7jTRPttKE32VQRw/fB8+zGY6cS0WYXpp13lYgwuzsTFRT+uKgtgipLLAQIPRRqU8umRoBd7CO0ILlgkpf1nUWU7+EdyB3YUQ9QuZLdhvxNIUnTK6cI9ozw7NMSLatysb+xsPCLUcgTYrlxB9uhzHZmmYnkK1JyPSg4dEG+Y6MKpeaqX5nOAfQuIfk4lVccivQtv70/cRycPrRYFuwLoU1qtKhtaLd8B88RY4U+NFtDuF/CVFqlNxFwTGxDV5TWljZzp4viiyspdvi3tje/ruFm+DRULrB1J0b8OrB3ZlnZWHcVC0yWKcnqKQo2z8ddws9KDA79icfk/hVNH/W6uMU30dH5k8vjj9+ozfYgXWCCn+qeHgVq+pCk2pHWRTV/JioZDrfnFHrMZvrQ6eKBqfZHdezQW9VWs7xK35I/46FMarCZQxyZBOYQqTxb9fddW1INHowAAAACV0RVh0ZGF0ZTpjcmVhdGUAMjAyMS0wMy0xOVQxMzowNTo0NiswMDowMMNJPDoAAAAldEVYdGRhdGU6bW9kaWZ5ADIwMjEtMDMtMTlUMTM6MDU6NDYrMDA6MDCyFISGAAAAAElFTkSuQmCC)datarootsdataroots

 ![](https://dataroots.io/static/658f5c0e02b7d8b175552e302f7d28ce/f3583/python-fabric-dwh.png)
](https://dataroots.io/research/contributions/connect-to-fabric-lakehouses-warehouses-from-python-code/?ref=dataroots.ghost.io)