Web Scraping’s Evolution: From Vintage Vibes to Futuristic Feels
Tags: tutorial, beginners, python, webdev
Published: 2023-10-19 17:28:51


![cats](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/wyzzi02rl8q9nxmjjyws.jpeg)
Hold onto your keyboards and brew that extra cup of coffee, because we're about to dive headfirst into the rollercoaster ride that is the evolution of web scraping in Python. From the primitive age of `sockets` to the futuristic promise of `httpx`, it's a tale of code, grit, and snake bytes!

###The Stone Age: Direct Sockets

Imagine a time when there were no libraries, just you, raw bytes, and a connection to the server.

```python
import socket

URI = "data.pr4e.org"
with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as s:
    s.connect((URI, 80))
    cmd = f"GET /romeo.txt HTTP/1.0\r\nHost: {URI}\r\n\r\n".encode()
    s.send(cmd)
    
    data = b''
    while True:
        response = s.recv(512)
        if len(response) < 1:
            break
            
        data += response

print(data.decode())
```

With `sockets`, you'd ride the raw frontier, establishing a connection straight from the saloon, sending messages to the server in Morse code (or what felt like it). It was raw, it was real, but boy, was it cumbersome.

### The Wild West: `urllib`

Fast forward, and we reach the Wild West of Python web scraping: the `urllib` era. Now, artists and coders alike didn't need to wrangle with sockets directly. A new dawn, a new way to fetch, and the world of web data was at our fingertips!

```python
from urllib import request

URL = 'http://data.pr4e.org/romeo.txt'
response = request.urlopen(URL)
data = b''
for line in response:
    data += line

print(response.headers.as_string())
print(data.decode())
```

No more Morse code – just the pure, distilled essence of web content, ready for digital Wild West saloon consumption. But wait, the saga continues...


Before we spotlight the upcoming third-party divas, let's craft our backstage hero, the `as_string` function. It'll elegantly choreograph those `headers` to ensure each starlet delivers a consistent, classic encore. 

```python
def as_string(headers:dict) -> str:
    """
    Pretty Prints Dictionary  
    """
    text = (' \n'.join(f'{key}: {value}' 
                 for key, value 
                 in headers.items()))

    return f"{text} \n\n"

```
### The Rockstar: `requests`

Enter the rockstar of web scraping: `requests`. With its leather jacket and cool demeanour, `requests` was like the guitarist who just knew how to shred:

```python
import requests

URL = 'http://data.pr4e.org/romeo.txt'
response = requests.get(URL)

print(as_string(response.headers))
print(response.text)
```

No more manual labor. Everything from session management to cookie handling was as effortless as playing an air guitar. Life was good, but the stage was set for the next sensation.

### The Futurist: `httpx`

In the neon glow of the future, where the world buzzes with asynchronous operations, `httpx` made its entrance, floating on the cloud of efficiency:

```python
import httpx

# response = httpx.get(URL) # synchronous 

# asynchronous magic
async with httpx.AsyncClient() as client:
    response = await client.get(URL)

print(as_string(response.headers))
print(response.text)
```

With both synchronous rhythms and asynchronous beats, `httpx` danced its way into the heart of developers, promising a brighter, faster web scraping future.

### Epilogue of Made-up Story 

So, there you have it: a tale of trials, triumphs, and transformations. From raw connections to refined libraries, web scraping in Python has been a journey worth every byte. As we look to the future, one thing's for sure: the adventure is far from over. 


### Output: `stdout.flush()`
The print outputs of all code above are like:
```sh

HTTP/1.1 200 OK
Date: Fri, 20 Oct 2023 08:20:33 GMT
Server: Apache/2.4.18 (Ubuntu)
Last-Modified: Sat, 13 May 2017 11:22:22 GMT
ETag: "a7-54f6609245537"
Accept-Ranges: bytes
Content-Length: 167
Cache-Control: max-age=0, no-cache, no-store, must-revalidate
Pragma: no-cache
Expires: Wed, 11 Jan 1984 05:00:00 GMT
Connection: close
Content-Type: text/plain

But soft what light through yonder window breaks
It is the east and Juliet is the sun
Arise fair sun and kill the envious moon
Who is already sick and pale with grief
```

___
Until then, keep on scraping and may your scrapes always return `200 OK`!