HTTP over unix sockets in Common Lisp
Tags: commonlisp, lisp, unixsocket, socket
Published: 2022-09-28 05:49:33

In this post we are going to take took at how we can send HTTP requests using Unix sockets in Common Lisp.

## Motivation
Let's first begin to analyze the motivation behind this post. To understand that, first we need to find an answer to the question, how do we send HTTP request using unix sockets in Common Lisp?

I have already asked the same question in the [Lisp Reddit Channel](https://www.reddit.com/r/lisp/comments/xd0sia/http_request_in_common_lisp_using_unix_socket/). This post is just an attempt to deep dive in to the answer to the problem at hand.

Basically all I want to do is to implement the following curl request in Lisp

```
curl --unix-socket /var/run/docker.sock http://localhost/v1.41/containers/json
```


## cl-docker

Why do I have a requirement like this in the first place? A couple of weeks back I have written a post about Running Docker commands in Lisp REPL where I have mentioned that the proper way to build a Lisp SDK for Docker involves sending HTTP requests using unix sockets. Hence this post is an elaborate explanation of how the solution actually works.

{% link https://dev.to/rajasegar/running-docker-commands-from-common-lisp-repls-350p %}

Before diving in to the solution, let's first set some context around Unix sockets, how it is different from TCP/IP sockets in the first place.

## Unix sockets
A UNIX socket, also known as UNIX Domain Socket, is an inter-process communication mechanism that allows bidirectional data exchange between processes running on the same machine. UNIX domain sockets know that they’re executing on the same system, so they can avoid some checks and operations (like routing); which makes them faster and lighter than IP sockets. So if you plan to communicate with processes on the same host, this is a better option than IP sockets.

UNIX domain sockets are subject to file system permissions, while TCP sockets can be controlled only on the packet filter level.

## The solution
This the answer I got from the user [Aidenn0](https://www.reddit.com/user/Aidenn0) that fits perfectly for my requirements.

```lisp
(let ((socket (make-instance 'sb-bsd-sockets:local-socket :type :stream)))
  (sb-bsd-sockets:socket-connect socket "/var/run/docker.sock")
  (let ((stream (sb-bsd-sockets:socket-make-stream socket
                                                   :element-type '(unsigned-byte 8)
                                                   :input t
                                                   :output t
                                                   :buffering :none)))
    (let ((wrapped-stream (flexi-streams:make-flexi-stream (drakma::make-chunked-stream stream)
                                                           :external-format drakma::+latin-1+)))
      (drakma:http-request "http://localhost/v1.41/containers/json" :stream wrapped-stream))))
```

Let's see how the above code works. 


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/o6gytmcs3ln4xxjt73re.jpg)

First, create a new socket by making a new instance of `sb-bsd-sockets:local-socket` with the socket type set as `:stream`.  And then make the socket connected to the local unix socket of Docker in `/var/run/docker.sock`. 


Now create a new socket stream and wrap it with the flexi-streams. 

## FLEXI-STREAMS
FLEXI-STREAMS implements "virtual" bivalent streams that can be layered atop real binary or bivalent streams and that can be used to read and write character data in various single- or multi-octet encodings which can be changed on the fly. It also supplies in-memory binary streams which are similar to string streams.

Finally pass this stream to the HTTP client library like Drakma to make the HTTP request using the wrapped stream. 


## Drakma
Drakma is a full-featured HTTP client implemented in Common Lisp. It knows how to handle HTTP/1.1 chunking, persistent connections, re-usable sockets, SSL, continuable uploads, file uploads, cookies, and more. 


This request will return a JSON response which can then be processed using any JSON library like [yason](https://xh4.github.io/yason/) or [cl-json](https://cl-json.common-lisp.dev/cl-json.html)


And this is how I used the solution to build an SDK for Docker in Common Lisp through the [cl-docker](https://github.com/rajasegar/cl-docker) package.

```lisp
(defun docker-api (url)
  (let ((socket (make-instance 'sb-bsd-sockets:local-socket :type :stream)))
	(sb-bsd-sockets:socket-connect socket "/var/run/docker.sock")
	(let ((stream (sb-bsd-sockets:socket-make-stream socket						
        :element-type '(unsigned-byte 8)
	:input t
	:output t
	:buffering :none)))
	(let ((wrapped-stream (flexi-streams:make-flexi-stream (drakma::make-chunked-stream stream)
	:external-format :utf-8)))
	(yason:parse (dex:get (docker-url url) :stream wrapped-stream :want-stream t) :object-as :alist)))))
```

FYI, I have used [Dexador](https://github.com/fukamachi/dexador) instead of Drakma to send HTTP requests by using the same stream object. The API of Dexador is pretty much similar to Drakma. And to parse the JSON response I use [yason](https://xh4.github.io/yason/) and returning the output as an Alist (association list).

## References:
- [Drakma](https://edicl.github.io/drakma/)
- [Flexi-streams](http://edicl.github.io/flexi-streams/)
- [Unix domain socket](https://en.wikipedia.org/wiki/Unix_domain_socket)
- [Unix sockets vs TCP/IP sockets](https://serverfault.com/questions/124517/what-is-the-difference-between-unix-sockets-and-tcp-ip-sockets)

Hope you enjoyed the post. I, myself had a lot of learning about sockets and streams, especially unix sockets while working on this problem. Please let me know for any queries/feedback in the comments section.