How to create a RESTful API to the PicoLisp Database
Tags: picolisp, lisp, database
Published: 2022-05-02 14:25:55

In the previous posts we have created a database and generated content that can be displayed in the browser. The format we chose was human-readable but not so suitable for automatic processing. 

Therefore we will now create a **REST-API** that enables other services to access the data in JSON format.

![outputjson.png](https://cdn.hashnode.com/res/hashnode/image/upload/v1634484153261/bkAOxT1VO.png)

-----------------------------

### What is a REST-API?

REST stands for **Re**presentational **S**tate **T**ransfer, which means that it fulfills the [following properties](https://en.wikipedia.org/wiki/Representational_state_transfer):
- client-server model, 
- stateless protocol (no session information needed),
- cacheability,
- layered system,
- uniform interface.

In simple words, what we want is to create an interface to our PicoLisp database that can be accessed from outside. The client doesn't have to speak PicoLisp, and the output format should be a standard machine-readable format such as JSON. 

Let's create an API that any client can access via a simple GET-request. Similarly to the previous example, the record is specified via the URL.

---------------------------

### Creating JSON output in PicoLisp

JSON (**J**ava**S**cript **O**bject **N**otation) is a very popular, language-independent data format that can be used to exchange data between different services.

It supports a number of data types:
- Numbers,
- Strings, delimited with double-quotes and backslash as escape symbol,
- Boolean (``true`` or ``false``)
- Arrays using square bracket notation with comma-separated elements,
- Objects as a collection of name-value pairs, where the names are also strings,
- ``null`` as empty value.

-----------------------

PicoLisp comes with a library to create json-output. It can be found in the library files under ``json.l``. There are four functions: 
- ``checkJson (X Item)``,
- ``parseJson (Str)``,
- ``readJson ()``,
- ``printJson (Item) ``.

For our purpose, we will only need the ``printJson (Item)`` function. Its functionality is very simple: it takes a [list of cons-pairs](https://picolisp-blog.hashnode.dev/concepts-and-data-types) and prints them in the JSON format.

Let's open the REPL (``$ pil +``) and try it at an example:

```
: (setq A 
   '( ( name . "Smith" ) 
   ( age . 25)
   ( address 
      ( street . "21 2nd Street") 
      ( city . "New York") 
      ( state .  "NY" ) 
      ( zip  .  10021 ) ) ) )
```

We have a nested list of cons-pairs, i. e. a list of cells where the "key" is found in the CAR and the "value" in the CDR. The keys are internal symbols (that's why it's ``name`` and not ``"name"``).

We can execute the ``printJson`` function on this:

```
: (printJson A)
{"name": "Smith", "age": 25, "address": {"street": "21 2nd Street", "city": "New York", "state": "NY", "zip": 10021}}
```

The function prints out a valid JSON string.

-------------------------

### Creating a cons-pair list

In order to be able to use ``printJson``, we need to convert the database output into a structured cons-pair list. This will require some manual formatting. Up to now, we only queried specific attributes for a record, like ``( : nm )``. However, we can also get a list of items using the ``getl`` function.

Let's test it in the REPL. We can start the database without the server using any of the previous scripts, for example [this one](https://gitlab.com/picolisp-blog/web-applications/-/blob/main/database/family-gui.l):

```
$ pil family.l -family~main +
family: 
```

It opens us a prompt directly in the family namespace. Let's get the item ``{A1}`` in list format:

```
family: (getl '{A1})
-> ((({A2} {A4}) . kids) ("Margaret Rose" . nm) ({A3} . mate) ({A11} . ma) ({A33} . pa) (705091 . dat) ("Countess of Snowdon" . job))
```

As you can see, it's a cons-pair list, but with some differences to our desired format:
1. Key and value are reversed. instead of ``( "Margaret Rose" . nm )`` we need ``( nm . "Margaret Rose" )``.
2. The symbol names at ``mate``, ``ma``, ``pa`` and so on should be replaced by the person's names.
3. The date ``705091`` should be formatted.

Let's go through it step by step.

----------------------

### Changing CAR and CDR in the cons pair

How can we convert  ``( "Margaret Rose" . nm )`` to ``( nm . "Margaret Rose" )``? Well, this is quite easy: We can build a new cons-pair using the ``cons`` function, which takes two arguments: CAR and CDR.

```
: (cons 1 2)
-> (1 . 2)
```

We can apply this to all items in the list using ``mapcar`` and an anonymous function:

```
(mapcar
   '((X)
      (cons (cdr X) (car X)) )
   (getl This ) 
```

Let's test it in the REPL:

```
family: (mapcar '((X) (cons (cdr X) (car X))) (getl '{A1}))
-> ((kids {A2} {A4}) (nm . "Margaret Rose") (mate . {A3}) (ma . {A11}) (pa . {A33}) (dat . 705091) (job . "Countess of Snowdon"))
```

-----------------------

### Formatting the list

To get a better overview, let's write down each cons-pair of the ``getl`` output: 

- ``(({A2} {A4}) . kids)``
- ``("Margaret Rose" . nm)``
- ``({A3} . mate)``
- ``({A11} . ma)``
- ``({A33} . pa)``
- ``(705091 . dat) ``
- ``("Countess of Snowdon" . job)``

Looking at the car, we have four cases: It is either a list of ``+Person`` objects, a number representing a date, a Person object, or a string. Let's treat either of these cases separately. We can switch between different "cases" using the ``cond`` function:

> ``(cond ('any1 . prg1) ('any2 . prg2) ..) -> any``  
> Multi-way conditional: If ``any`` of the ``anyN`` conditions evaluates to non-``NIL``, ``prgN`` is executed and the result returned. Otherwise (all conditions evaluate to ``NIL``), ``NIL`` is returned.

----------------------

**Case 1: The CAR is a number.**

If the value (let's call it ``V``) is a number, return the formatted value. We can test whether it's a number with the ``num?`` function.

```
(cond
   ((num? V) (datStr V))
```

---------------------

**Case 2: The CAR is an ``+Person`` object.**  

To check this, we can use the ``isa`` function, which takes a class and an object. If yes, we want to get the name property of this object. We can get it with the ``;`` function:

```
(cond
   ((num? V) (datStr V))
   ((isa '+Person V) (; V nm))
```
-----------------------

** Case 3: The CAR is a list of ``+Person`` objects.**

Now this one is a little bit tricky. Let's say the CAR is a list. In this case, we want to loop over every list item and return the name. We can do this with ``mapcar`` and an anonymous function which takes an object and returns it name property:

```
(mapcar '((This) (: nm)) V)
```

But how can we check if ``V`` is a list? For this purpose, we can use the ``pair`` function which checks if the argument is a cons pair and returns it if true. Technically, a "normal" list is also a cons pair, while numbers, strings and and objects (i. e. all the other cases) are not. So we can expand our ``cond`` condition list with the following line:

```
(cond
   ((num? V) (datStr V))
   ((isa '+Person V) (; V nm))
   ((pair V) (mapcar '((This) (: nm)) V) )
```


-----------------------

** Case 4: The CAR is  a string.**

Lastly, if the CAR is a string or any other case we didn't consider, we do nothing with it:

```
(cond
   ((num? V) (datStr V))
   ((isa '+Person V) (; V nm))
   ((pair V) (mapcar '((This) (: nm)) V) )
   (T V) )
```

------------------------

### Bringing it together

Now we combine everything in one function:
1. get the list with ``(getl This)``
2. apply ``mapcar`` to build a new ``cons`` pair
3. before we set the ``car``, we modify it depending on the result of ``cond``.

```
(mapcar 
   '((X)
      (cons (cdr X) 
         (let V (car X)
            (cond
               ((pair V)
               (mapcar '((This) (: dat)) V) )
               ((num? V) (datStr V))  # Can only be date
               ((isa '+Person V) (; V nm))
               (T V) ) ) ) )
   (getl This) ) 
```

----------------------

### Convert to JSON and return it!

Now that we have our list, all we have to do is converting it and modify our HTTP-Header so that it returns ``Content-Type: application/json`` instead of ``text/html``. Instead of our standard ``html`` function, we call the function ``httpHead`` with the arguments ``"application/json"`` for the content-type and ``0`` for the cache-control. Let's test it in the REPL:

```
family: (httpHead "application/json" 0)

HTTP/1.0 200 OK
Server: PicoLisp
Date: Sun, 17 Oct 2021 14:44:31 GMT
Cache-Control: max-age=0
Cache-Control: private, no-store, no-cache
Content-Type: application/json
```

The function returns a complete header which informs the browser that the received data format is JSON. 

-------------------

Now we have to send it. In the HTTP protocol, there are two possibilities to upload data: Either the content-length is communicated in the header (which is difficult, i. e. expensive in a dynamically created page), or the upload is [sent in chunks](https://en.wikipedia.org/wiki/Chunked_transfer_encoding). We can create chunked upload and send the content via the ``ht:Out`` function from the ``ht``library.

```
   (httpHead "application/json" 0)
   (ht:Out *Chunked
      (printJson
         ...
```

As final step, we wrap it in a function called ``person.json``. The ``.json`` is not a requirement, but it makes clear that the output of this format is a json-format. Accordingly, we modify the ``server`` and the ``allowed``-function. 

```
(allowed NIL
   "@lib.css" "!person.json" )

...

(de person.json (This)
   (httpHead "application/json" 0)
   (ht:Out *Chunked
      (printJson
         ...
   ...
```


Then we can start the program with

```
$ pil family-rest.l -family~main -go +
```

If we now point the server towards http://localhost:8080/?-A67, we see the following JSON-formatted output:

![outputjson.png](https://cdn.hashnode.com/res/hashnode/image/upload/v1634484153261/bkAOxT1VO.png)

This output can be read by any application that calls a GET request to the URL.

----------------------

That's it! The complete source code to this example can be found [here](https://gitlab.com/picolisp-blog/web-applications/-/blob/main/database/family-rest.l).

-----------------------

### Sources

- https://software-lab.de/doc/index.html  
- https://software-lab.de/doc/app.html