Simplifying Data Analysis With AI and DevTools - for the non-Python developer!
Tags: python, ai, beginners, thisweekinai
Published: 2024-02-04 17:17:12

---
title: Simplifying Data Analysis With AI and DevTools - for the non-Python developer!
published: true
description: As an app developer you've probably generated (or accessed) a lot of data, and wished you had expertise to analyze and visualize the results. But you're not a Python developer! How can you get productive quickly? In this @PyDataNYC 2023 talk, I show you 4 tools and a process to jumpstart your data science journey.
tags: python, ai, beginners, thisweekinai
cover_image: https://dev-to-uploads.s3.amazonaws.com/uploads/articles/4and0vjp8li5c3in78bt.png
series: This Week In AI News
---


**3 Resources To JumpStart Your DataScience Journey:**
1. [Data Science For Beginners Curriculum](https://microsoft.github.io/Data-Science-For-Beginners/#/) - learn the fundamentals!
3. [Data Science In Visual Studio Code](https://code.visualstudio.com/docs/datascience/overview) - learn to use core tools!
2. [PyDataNYC 2023 Workshop](aka.ms/pydata-workshop-2023) - fork it, and bring your own dataset!

---

Welcome to the fourth post in my [**This Week in AI News**](https://dev.to/nitya/series/25819) series! Today, I want to talk about Data Science - but first, some context.

## 1 | 2024 Resolution: Weekly Blog Series

Like many of you, I've had an aspirational goal to blog more. But finding time to plan content and publish it regularly, was hard. Then I joined the AI Advocacy team and found myself learning, building, and collaborating on, a number of interesting tools, technologies and projects in AI, ML and Data Science. Content problem solved. 😍

Now all I needed was accountability. Say hello to [This Week in AI News](https://dev.to/nitya/series/25819). In 2024, my resolution is to write one a week - with a deep dive into 1 topic at a time. Want more ways to learn?
- [Join AI Developers & Entrepreneurs](https://meetup.com/aide-hub) 👉🏽 our in-person meetup in NYC
- [Follow My LinkedIn feed](https://www.linkedin.com/in/nityan/) 👉🏽 get AI news & updates as they happen

---

## 2 | This Week: Skilling Up On Data Science

As AI enters every aspect of our daily lives, it becomes useful (if not necessary) to build skills in both data science and app development. **And you don't need to be a Python or Data Science expert to become productive**! Instead, take advantage of core developer tools and AI assistance, _and build your knowledge and expertise by playing with real data_. That was core message in my PyData NYC 2023 talk. 

{% twitter 1720052414491861342 %}

I had two core objectives:

1. **Focused Learning**. Prioritize learning just want you need to learn, to get you closer to completing the targeted task. Don't boil the ocean. Optimize your time.
1. **Transferable Learning**. Use the right tools and techniques to allow your learning to be _shared with others_, making it easy for you to collaboratively debug issues with experts (now) and reuse/extend as templates (later).

And my philosophy was simple. _Don't try to learn everything_. You cannot match the knowledge of those with decades of experience. Instead, focus on _solving a problem_ so your knowledge is tied to practical usage.

---

## 3 | Resource: PyData NYC Workshop

Want to explore this further? First, here are 3 links to bookmark and revisit, from my PyDataNYC talk:

### 3.1 Video Recording

Watch the talk replay on YouTube here: 

{% youtube wVMwKy1QmTM %}

### 3.2 Presentation Slides

Flip through the slides as you watch, to get better context.

{% speakerdeck 4c3daeda993f4a46836ac3251950c42b %}

### 3.3 Workshop Repo

Fork [this repo](https://aka.ms/pydata-workshop-2023) and follow along if you meet the pre-requisites. _Note_: The repo will be getting actively updated over the next few weeks with more exercises. The PyData NYC version will remain archived at the [pydata-nyc-2023](https://github.com/nitya/pydata-workshop-2023/tree/pydata-nyc-2023) branch.

![Screenshot with details of workshop](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/08jnksk1ne3l2nn16vo6.png)

---

## 4 | Overview: What You'll Learn

By the end of that workshop / talk, you should be able to:
1. Use Jupyter Notebooks - and explain how they help with transferable learning.
2. Use GitHub Codespaces and explain how they simplify setup with prebuilt dev environments.
3. Use GitHub Copilot - and explain how it supports focused learning with context-driven suggestions.
4. Analyze and Visualize Data - using an open-source dataset and traditional tools & techniques.
5. Grow your Data Intuition - using Project LIDA (with OpenAI) to define goals and get suggestions.

The last two steps are particularly important to differentiate because data science is still more of an art than a science. Given a dataset, _extracting insights_ can have two paths:
1. **Known questions**. The first is where you want to use the data to answer a specific question - and you just need help with the right tools & process to get there.
2. **Unknown questions**. The second is where you don't know what questions to ask in the first place - and you need to start building up your intuition on how to define "what insights are possible" with a given dataset.

---

## 5 | Exploratory Data Analysis: IPL 2022

The first step is to do _exploratory data analysis_ (EDA) with the given dataset, using traditional tools and techniques to get preliminary insights. For [my talk](https://aka.ms/pydata-workshop-2023) I used a Kaggle IPL 2022 dataset for 3 reasons:

1. Kaggle datasets provide a [Data Card](https://www.kaggle.com/datasets/aravindas01/ipl-2022dataset/data) that gives us details on the data structure, origins and usage requirements.
2. The Kaggle community publishes sample EDA notebooks [like this IPL 2022 example](https://www.kaggle.com/code/hashimali179/ipl-2022-analysis) that can provide learning inspiration.
3. You can now build on your learning journey by trying new datasets and exploring more EDA notebooks - with the same development environment, as a template.

![IPL 2022 Dataset](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/2xkzdw228x7sbnq7n1xm.png)

How do you get started with your **exploratory data analysis** (EDA) task? Look at EDA examples and see how you can replicate those steps on your own, using the GitHub Codespaces (pre-built environment) and GitHub Copilot (code explainers coaching) to streamline learning.

![Can I have it visualize something specific](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/6gvirq88eg57ecj5dzu6.png)

What if you didn't have example EDAs to learn from? How can you build your own intuition for what you can do? Let's talk about Project LIDA from Microsoft.

## 6 | LIDA: Generate Data Visualizations with AI

[Project LIDA](https://aka.ms/lida/github) is a Microsoft Research open-source project that can _automatically generate_ data visualizations from your dataset, using the power of Large Language Models. The screenshot below shows the approach LIDA takes - convering a data set into a compact _natural language representation_ (context) that can be used with a variety of Large Language Model providers (OpenAI, Azure OpenAI, HuggingFace) and data visualization libraries (matplotlib, d3, seaborn etc.)

![LIDA Goals](https://github.com/microsoft/lida/raw/main/docs/images/lidamodules.jpg)

Most importantly, it allows me (as a developer) to focus on defining my task using natural language prompts, with appropriate system contexts, to not just automate the visualization - but also get suggestions that build my intuition.

For example: I can **specify a persona showing interest in a specific team (Mumbai Indians)** resulting in the data visualization goals being auto-generated for me to derive insights relevant to that team.

![Project LIDA](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/iv3gntnvo6lfso5at0cp.png)

And I can tailor my prompt _using natural language to tailor the underlying code libraries_ in a more intuitive way without having to know the deeper details of the Python APIs or functionality. 

![Color Code](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/4a017hc4xnkv4espllu0.png)

If you find this interesting, check out the [LIDA Project](https://aka.ms/lida/org) organization on GitHub for some side-projects using LIDA - include a Codespaces template that you can instantiate and use for your own experiments.

## 7 | Visual Studio Code: For Data Science

We covered a lot, but the main goal here was to help you realize how much _developer tools and AI_ can help you with jumpstarting your developer journey into data science, even as a non-Python developer. Every learning journey starts with setting up a development environment that empowers you - and I love Visual Studio Code. So here are two resources I recommend, to help you get started:

### 7.1 | Data Science Tutorial With VS Code

This [Visual Studio Code series](https://code.visualstudio.com/docs/datascience/overview) will teach you everything you need to know, to get started using VS Code for your data science development - from basic Notebooks, to training and deploying machine learning solutions on Azure.

![Data Science Tutorial With VS Code](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/yd8vvuuz1v942mje79bs.png)

### 7.2 | Data Wrangler Extension for VS Code

[Data Wrangler](https://aka.ms/datawrangler.) is a code-centric data cleaning tool that is integrated into VS Code and VS Code Jupyter Notebooks - and was [released in early 2023](https://devblogs.microsoft.com/python/data-wrangler-release/) as a tool for data scientists to speed up the data preparation & analysis process. It does this by also providing a rich user interface that automatically generates Pandas code to help show insightful data visualizations.

For example: In our IPL 2022 analysis, we could have run this tool early, to identify how representative the data was in terms of coverage across all features and contexts for our analysis. For a step-by-step tutorial on using Data Wrangler, check out [their docs](https://github.com/microsoft/vscode-data-wrangler).

![Data Wrangler Extension for VS Code](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/wq6u1f97c3nzjmlw51qm.png)

---

## Summary

That was a lot!! But hopefully it inspires you to go out and grab a dataset and start your own data analysis journey. Watch this space for an updated workshop and tutorials that will showcase a few more data science tools and tips later this year. 

**3 Resources To JumpStart Your DataScience Journey:**
1. [Data Science For Beginners Curriculum](https://microsoft.github.io/Data-Science-For-Beginners/#/) - learn the fundamentals!
3. [Data Science In Visual Studio Code](https://code.visualstudio.com/docs/datascience/overview) - learn to use core tools!
2. [PyDataNYC 2023 Workshop](aka.ms/pydata-workshop-2023) - fork it, and bring your own dataset!

And follow the Azure org for more articles from the Developer Relations team.

{% org azure %}