Let's Build HTTP Client From Scratch In Rust
Tags: rust, dns, http
Published: 2024-06-13 21:45:26

[Previously](https://dev.to/hiro_111/lets-build-http-parser-from-scratch-in-rust-5gih), we dived into how to make an HTTP parser with no dependencies. In this blog let's extend our knowledge further by building an HTTP client CLI😉

(I call our brand-new CLI [`fetch`](https://github.com/hirotake111/rust_fetch), by the way)

It would be very time-consuming if we implemented all the feature requirements for modern HTTP client CLI. Instead let’s just build a simple one that only supports HTTP/1.1 protocol using IPv4 Address (no HTTP/2, 3, and IPv6. Also some features in HTTP/1.1 will be missing).

## API Design

Let's first think about how our brand-new CLI will be used. It should be similar to a well-known tool curl. But I like to have an HTTP method as an argument explicitly.
Also we may want to POST data to a server using HTTP body (or not). So we have possibly three arguments passed to the CLI:

- HTTP method
- URL
- HTTP body (Optional)

And here is an example command.

```bash
fetch post example.com '{"foo": "bar"}'
```

Now we can see what our internal API look like. Let's write our `main` function:

```rust
fn main() {
    let mut args = std::env::args();
    let program = args.next().unwrap();
    if args.len() <= 1 {
        display_usage(&program);
        exit(1);
    }
    let method: Method = args.next().unwrap().parse().unwrap();
    let url = args.next().unwrap();
    let body = args.next();
    let client = Client::new(); // Initialize a client
    // Make an HTTP request
    let response = client.perform(method, url, body).unwrap();
    println!("{response}");
}
```

That's enough now.

## DNS Client

Now let's write our client struct and its corresponding method called `perform`:

```rust
pub struct Client {};

impl Client {
    pub fn new() -> Self {
        Self {}
    }
}

impl Client {
    pub fn perform(
        &self,
        method: Method,
        url: String,
        body: Option<String>,
    ) -> Result<String, String> {
        let (protocol, url) = url.split_once("://").unwrap_or(("http", &url));
        let (hostname, url) = match url.split_once('/') {
            Some((hostname, url)) => (hostname, format!("/{url}")),
            None => (url, "/".to_string()),
        };

        // ...!?
    }
}
```

Finally, I realized we need a way to resolve DNS in our project😓 Don't worry! I have implemented it [before](https://dev.to/hiro_111/building-a-simple-dns-client-in-rust-5bp0).
Let's just copy-and-paste the code for the DNS client from my repository.

## HTTP Request

Now we have an HTTP method, a server’s hostname, a URL path, a payload to be sent, and a server’s IP address.

So what to do next? Well we need to create our HTTP request out of some parameters above. Let’s do it.

```rust
let request = HTTPRequest::new(method, hostname, &url, body);
```

In short, our HTTP Request struct looks like below:

```rust
#[derive(Debug, Clone)]
pub struct HTTPRequest {
    request_line: RequestLine,
    headers: HTTPHeaders,
    body: Option<String>,
}

impl HTTPRequest {
    pub fn new(method: Method, hostname: &str, url: &str, body: Option<String>) -> Self {
        let request_line = RequestLine::new(method, url);
        let headers: HTTPHeaders = vec![("Host".to_string(), hostname.to_string())].into();
        Self {
            request_line,
            headers,
            body,
        }
    }
}
```

Once we create an HTTP request, we can serialize and send it to the server via TCP stream:

```rust
// Connect to a server
let mut stream = match protocol {
    Protocol::HTTP => TcpStream::connect((addr, 80)).map_err(|e| e.to_string())?,
    Protocol::HTTPS => unimplemented!(),
};
// Send an HTTP request to the server
let request = HTTPRequest::new(method, hostname, &url, body);
let n = stream
    .write(request.to_string().as_bytes())
    .map_err(|e| e.to_string())?;
println!("sent {n} bytes");
```

## HTTP Response

Finally, we can receive HTTP response from the server, and make something processable (our HTTP Response) out of it:

```rust
// After sending HTTP request, create a buf reader and get data in it
let reader = BufReader::new(stream);
let response = HTTPResponse::try_from(reader)?;
println!("{:?}", response);
```

And our HTTPResponse struct should be like below:

```rust
#[derive(Debug, Clone)]
pub struct HTTPResponse {
    status_line: StatusLine,
    headers: HTTPHeaders,
    body: Option<String>,
}

impl<R: Read> TryFrom<BufReader<R>> for HTTPResponse {
    type Error = String;

    fn try_from(reader: BufReader<R>) -> Result<Self, Self::Error> {
        let mut iterator = reader.lines().map_while(Result::ok).peekable();
        let status_line: StatusLine = iterator
            .next()
            .ok_or("failed to get status line")?
            .parse()?;
        let headers = HTTPHeaders::new(&mut iterator)?;
        let body = if iterator.peek().is_some() {
            Some(iterator.collect())
        } else {
            None
        };
        Ok(HTTPResponse {
            status_line,
            headers,
            body,
        })
    }
}
```

We are done! Unfortunately, not really.

## Test

When I `cargo run` it, I found the program doesn’t finish after receiving an HTTP response from the server.

```bash
cargo run -- get example.com
// -> this just blocks and never ends...
```

Why? Our HTTPResponse parser implementation worked in the previous project. So it's supposed to work this time too...

Well, it turned out, in my previous project I tested my HTTPResponse parser using data in a text file. However, in read world HTTP response there is no `end of file` section. So it turns out we need somehow stop reading the byte stream when we find an `empty line`.

Here is updated version of my implementation:

```rust
impl<R: Read> TryFrom<BufReader<R>> for HTTPResponse {
    type Error = String;

    fn try_from(reader: BufReader<R>) -> Result<Self, Self::Error> {
		    // ...
        let mut body = vec![];
        for data in iterator {
            // Break if it's just an empty line
            if data.is_empty() {
                break;
            }
            body.push(data);
        }
        Ok(HTTPResponse {
            status_line,
            headers,
            body: Some(body.join("\n")),
        })
    }
}
```

Then, let’s try it again.

```bash
cargo run -- get example.com
// -> works!
```

Congratulations!! 

However here is another challenge - when I run it for other URL (say, google.com) it again doesn’t finish…

And this is the biggest lesson learned in the project for me.

## **HTTP/1.1 Persistent Connections**

By default, HTTP/1.1 uses persistent connections. This means a single TCP connection can be used to send and receive multiple HTTP requests and responses. This improves efficiency by avoiding the overhead of re-establishing connections for each request.

Many servers, including `google.com`, use chunked transfer encoding for the body. This encoding allows the server to send the body in chunks of variable size, with each chunk preceded by its size information.

Simply put - some HTTP servers don’t even have an empty line in the HTTP body.

My idea after understanding the cause of the issue was, to read the `Content-Length` HTTP header, and then read the exact bytes from the byte stream.

So here is my final implementation of HTTPResponse struct:

```rust
impl<R: Read> TryFrom<BufReader<R>> for HTTPResponse {
    type Error = String;

    fn try_from(reader: BufReader<R>) -> Result<Self, Self::Error> {
        // The use of .lines() splits the stream by new line (\n, or \r\n).
        // But this makes it impossible to parse HTTP body for us.
        // So instead, leverage .split(b'\n')
        let mut iterator = reader.split(b'\n').map_while(Result::ok).peekable();
        let status_line: StatusLine = iterator
            .next()
            .ok_or("failed to get status line")?
            .try_into()?;
        let headers = HTTPHeaders::new(&mut iterator)?;
        // The length of the HTTP body
        let mut length = headers
            .0
            .get("Content-Length")
            .ok_or("HTTP header doesn't have Content-Length header in it")?
            .parse::<usize>()
            .map_err(|e| e.to_string())?;
        let mut body = vec![];
        for mut data in iterator {
            data.push(b'\n');
            length -= data.len();
            body.push(data);
            if length <= 0 {
                break;
            }
        }
        let body = body.into_iter().flatten().collect::<Vec<u8>>();
        let body = String::from_utf8(body).map_err(|e| e.to_string())?;
        Ok(HTTPResponse {
            status_line,
            headers,
            body: Some(body),
        })
    }
}
```

And here is the final result:

![final result of our HTTP client CLI](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/jwv9fh1r2ly4iia2b59y.png)

🎉

Thanks for reading 😉