Web Scraping 101
Tags: webdev, showdev, devops, beginners
Published: 2021-03-03 08:42:31

What is Web Scraping?
Ans:  Try [Google Search](https://www.google.com/search?q=web+scraping&rlz=1C1CHBD_enIN942IN942&oq=Web+&aqs=chrome.0.69i59l3j69i60l3j69i65l2.2099j0j1&sourceid=chrome&ie=UTF-8) lol.

As the name suggests it's just scraping data of the web sites using http request response cycle.
Request response cycle is just two way communication between servers or computers browser is just an interface like messanger and servers are like users.
Fun tip!!!
Try writing 
<code>curl https://www.example.com</code>
in your native terminal.
That's also web scraping of sort. 

But let's just cut to the chase.

So today we are gonna be scraping all the image from udemy.com, sounds fun :o!!!

To get started first you need to install a couple of python modules 
<em>requests, bs4</em> and <em>lxml</em>

So <u>requests</u> package is used to make requests to the server,
<u>bs4</u> is a tool that make stuff looks pretty and easy to work with,
and <u>lxml</u> well that's a parser to parse HTML response markup and goes along with bs4.BeautifulSoup() like a sauce.

We're ready and good to go.

Now just follow along. 

Step 1: Import all the necessary modules.
<img src="https://dev-to-uploads.s3.amazonaws.com/uploads/articles/a4znrv7cmepir5aey3pn.png" alt="importing stuff">

Step 2: Declare variables <u>name</u> and <u>url</u> to make stuff easy.(totally optional)
<img src="https://dev-to-uploads.s3.amazonaws.com/uploads/articles/yqe1ddj9bnrk7pqq20v4.png" alt="declaring variables">

Step 3: Request server for HTML code via url and create a soup object of retrieved data via parser and bs4.BeautifulSoup() method.
<img src="https://dev-to-uploads.s3.amazonaws.com/uploads/articles/zomuh7kfd77v3nleh46r.png" alt="requesting code and soup">

Step 4: Select and fetch all the image objects or nodes.
<img src="https://dev-to-uploads.s3.amazonaws.com/uploads/articles/qrbrfb5pk44xw7jvuy5y.png" alt="fetching images">

Step 5: Parse all the images nodes one after the other to get image source which will help us to make requests for image data. 
<img src="https://dev-to-uploads.s3.amazonaws.com/uploads/articles/mu79ygz8kxm21nwlhyda.png" alt="parsing images">

Step 6: Select source(value of src attribute) of the image.
At this time if you will try to print an image object it will look exactly like corresponding html code. 
<img src="https://dev-to-uploads.s3.amazonaws.com/uploads/articles/l51hc4fsgx4rissp8223.png" alt="store url">
 
Step 7: Make a request to the corresponding server to receive image data.
So now we have image source, which can be used to in request.get() to ask some other server for "The Image Data". But we got to check for the type first because in HTML5 we can also use svg as source of an image that may be represented differently than the image. In short, we can surely save an svg's via this method but I just haven't added that functionality yet!
In fact you can fetch any type of data GIF's, PNG's, JPG's, you name it.  
<img src="https://dev-to-uploads.s3.amazonaws.com/uploads/articles/qmibyidlohx4ot470vlt.png" alt="get image">

Step 8: Open a file in binary format to save data.
<img src="https://dev-to-uploads.s3.amazonaws.com/uploads/articles/n462gum2v7xn6dysf1ue.png" alt="open file">

Step 9: Write all the data received from imageNode.content attribute.
<img src="https://dev-to-uploads.s3.amazonaws.com/uploads/articles/5cjzfy101xkh9p8qc1an.png" alt="write data">

Step 10: Bruh, you did it ! Congrats!!!

Let the loop complete its job.
Now you have a new script to show off.

All the code at one place:
<a href="https://gist.github.com/icecoffee-dev/8ac097210fbc05878d1300fb735680cb" target= "_blank">
<img src="https://dev-to-uploads.s3.amazonaws.com/uploads/articles/38fg68igpw99d79af6s2.png" alt="code">
</a>

You did a great job. Now you can scrape any type of data you like from a normal website. 

<strong>What to do Now???</strong>
Try reading documentation of [beautifulsoup4](https://pypi.org/project/beautifulsoup4/), [requests](https://pypi.org/project/requests/), [lxml](https://pypi.org/project/lxml/) or any other url parser you liked this stuff.

Before that, there is one more thing.
You can also fetch image file names from those source url's.
I prefer to do that by regex and believe me it's weird to see those funny looking patterns at first so please don't judge .

##Exercise
{% post https://dev.to/icecoffee/example-domains-4i54 %}
Goto some site like the one displayed above and find out number of words in the page. 
I'd love to hear about your progress in the comments section.



Hope, you liked it.
Embrace it's power.
<img src="https://c.tenor.com/rNZG_zJFO1AAAAAM/yoda-star-wars.gif" alt="Power" />
and make some thing beautiful.


Have a beautiful day. ✨

<strong>Resources: </strong>
[Cover image](https://images.app.goo.gl/93AcFeJ3uE19Z4z88)