Why I HAD To Scrape The Web For Some Data
Tags: career, programming, beginners, webscraping
Published: 2022-04-03 16:20:46

[Show me the conclusions IDC](#conclusions)

It all started on a cold night.....JK 😉

It all started when I decided that I want to make my own project so that I will check if developing websites is something that can interest me.

But I didn't want to make a `todo` or a `movie` site, I wanted to do it about something that I can connect with.

I thought: _if I'll feel connected to the topic of the project it will be a good foundation for the motivation I'll need to harness for this mission._

And...**I was right** [insert self pat on the back emoji here]


So the first thing that popped in my head was PLANTS! 🌱

Why? 'cause I've always liked growing them! 

Ok...everything was going according to plan.
I decided the project will be an Ecommerce store for plants.

The name of the store is _grow._ ... yes, I thought about it myself... thank you.

![yes I did](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/4f390nzs1d9e3yniuj3d.gif)
 

The only thing that was missing was the data.
I thought: "_No problem, I'm sure there are a tone of APIs waiting for me in the world wide web_" 

Little did I know **I was wrong**, all I could find was a bunch of photos of roots for Machine Learning projects to Identify a sick plant, which don't get me wrong, sounds really cool, but I needed DATA.

At the time, I looked at a lot of online plant stores to draw some inspiration.
And then it came to me! I'll just take the plants from one of the websites I found!

![Genius](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/kyyk773o5xzf8yvk640d.jpeg)
 

But that will be too tedious to do one by one....
**So I'll throw some code at this problem and build a bot to scrape it for me!**

I searched how to do that,
that's where I found out it's called 🔥**Web Scraping**🔥.

> **What's web scraping?**
> Web scraping is the process of <u>extracting data</u> from a website in an <u>automated fashion</u>.
> ![Flow of Scraping The Data](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/fkxcaaayvam9dq6aq0nb.png)


After a small search I came across an article that suggested `Selenium` and `python`. (in hindsight they weren't the best tools for the job, but it was still fun learning about this whole new world called web scraping).



**OK OK STOP! Where is the code though?**

The Bot's Code can be found [here](https://github.com/danielbellmas/plant-shop/blob/master/server/houseplant-bot.py) (it's not perfect but it did the job).

The full project can be found on [my github](https://github.com/danielbellmas/plant-shop) ⭐️


Here is a little snippet of the scrape :
[Snippet of the scrape](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/o84rqoa3erfq66gglcwb.gif)


This is how I felt when I saw my script moves through the pages by itself - [invincible](https://www.youtube.com/watch?v=g0YE9Ofww7s&ab_channel=LudwinClips)



### ** All of the scraped data is for personal use only, and no plants were harmed in the process.

---

### Conclusions:

- Choose a topic **that you enjoy** for your personal projects, it'll take you far.

- **Don't stress** over the technologies, if you enjoy programming you'll enjoy and learn from every project (so much!)

- There will always be some obstacles in the way of creating something, our job is to **make it happen** nonetheless.


![Don't forget to have fun!](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/mfxl2hqki6p1d1h3ydfk.jpeg)