Searching, Sorting and Pagination in a Common Lisp web application
Tags: commonlisp, webdev, lisp
Published: 2021-07-14 05:42:12


In this post, we are going to build a demo web application in Common Lisp with search, sort and pagination functionalities with a tabular data of randomly generated food dish names, ratings, price and cuisines.


# Project bootstraping

As usual we are going to use [Caveman](https://github.com/fukamachi/caveman) for scaffolding our web application. If you want to know more about Caveman or why Caveman, you might want to take a look at my previous [post](https://hangaroundtheweb.com/posts/lisp-for-the-web-5)
about Common Lisp and web development. Caveman is available in quicklisp, so you can install it with:

    (ql:quickload :caveman2)

And you can start a create a new project with Caveman like this:

    (caveman2:make-project #P"~/quicklisp/local-projects/cl-tabular" :author "Rajasegar")


# Index route

Now take a look at our `index` route. Our index route is going to use some query params for sorting and pagination.
With Caveman you can parse the query params using the `_parsed` key and we are using custom defined function to 
take the query parameter values from it like below:

```lisp
    (defun query-param (name parsed)
      (cdr (assoc name parsed :test #'string=)))
```

So for our route logic, we need the values of the following query parameters, start which is the starting offset
of the records for the page, direction which is either ascending or descending and the sort-by key based on which
column we are currently sorting the list.
```lisp
    (defroute "/" (&key _parsed)
      (format t "_parsed = ~a~%" _parsed)
      (let ((start (parse-integer (or (query-param "start" _parsed) "0")))
    	(direction (or (query-param "direction" _parsed) "asc"))
    	(sort-by (or (query-param "sort-by" _parsed) "name")))
        (render #P"index.html"
    	    (list
    	     :foods (slice-list start (sort-list direction sort-by))
    	     :total (length *foods*)
    	     :pages (generate-pages)
    	     :start start
    	     :direction direction
    	     :sort-by sort-by
    	     :opposite-direction (get-opposite-direction direction)))))
```
One thing important to note in the above code snippet is how we are sending data to the templates via the `render` function. We can construct our data using a list and the templates can easily access them via the keyword mapped to the data. 

Say, for example, in the template you can loop through the list of foods like
```lisp
    {% for food in foods}
    <p>{{food.name}}</p>
    {% endfor %}
```
So how are we sending the paginated list of foods to our home page. If you take a closer look, we are first sorting the entire list of foods using the `sort-by` parameter and then 
we are slicing the list based on the `start` offset and returning them to the template.
```lisp
    :foods (slice-list start (sort-list direction sort-by))
```
Let's take a look at our `slice-list` function on how we are slicing our list.
```lisp
    (defun slice-list (start)
      (let ((new-list nil))
        (dotimes (i 10)
          (push (elt *foods* (+ i start)) new-list))
        new-list))
```
We construct a new temporary list by pushing the 10 items from the original list, starting from the `start` offset and then returning the new list to the page. The `sort-list` function is discussed in the later part of the post under Sorting.


# Index template

Our index template is very big and has got three sections, the search form, the table and the pagination.


## Search Form

First we will take a look at the search form. So this is a simple form with an input field with the name `query`.
```html
    <form>
      <div class="mb-4">
        <div class="col-6">
          <input 
    	  class="form-control form-control-lg" 
    	  type="text" 
    	  placeholder="Search dish name..."  
    	  name="query" 
    	  hx-post="/search?start=0&direction=asc&sort-by=name" 
    	  hx-trigger="keyup changed delay:500ms" 
    	  hx-target="#results">
        </div>
      </div>
    </form>
```
We are adding some custom attributes starting with `hx-` , these are actually some enhanced attributes for HTML using a library called [htmx](https://htmx.org) which  allows you to access AJAX, CSS Transitions, WebSockets and Server Sent Events directly in HTML, using attributes, so you can build modern user interfaces with the simplicity and power of hypertext

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/9mqah4xoql5u8uozomli.png)


Using htmx, we are sending a POST request to the url `/search` and the response will be swapped with the element with an id `#results`. Hence as soon as you start typing the search keywords the client will start sending the request to the server with a delay of 500 milli seconds and you get to see the results populated in the table.


## Table

Next comes our important UI component the table itself. The table will have four columns like Name, Rating, Price and Cuisine. When you click on the table headers, it will toggle the sort direction automatically from ascending to descending and vice-versa for a particular column value. We will show some up and down arrows to indicate the sorting direction.

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/fgmx86d8gn7cv7sw7vtk.png)


```html
    <div id="results">
      <p>{{total}} results found</p>
      <table class="table table-striped">
        <thead>
          <tr class="table-dark">
    	<th><a href="/?start=0&sort-by=name&amp;direction={{opposite-direction}}">Name
    	    {% if sort-by == "name" and  direction == "asc" %} &uarr; {% endif %}
    	    {% if sort-by == "name" and  direction == "desc" %} &darr; {% endif %}
    	</a></th>
    	<th><a href="/?start=0&sort-by=rating&amp;direction={{opposite-direction}}">Rating
    	    {% if sort-by == "rating" and  direction == "asc" %} &uarr; {% endif %}
    	    {% if sort-by == "rating" and  direction == "desc" %} &darr; {% endif %}
    	</a></th>
    	<th><a href="/?start=0&sort-by=price&amp;direction={{opposite-direction}}"> Price
    	    {% if sort-by == "price" and  direction == "asc" %} &uarr; {% endif %}
    	    {% if sort-by == "price" and  direction == "desc" %} &darr; {% endif %}
    	</a></th>
    	<th><a href="/?start=0&sort-by=cuisine&amp;direction={{opposite-direction}}">Cuisine
    	    {% if sort-by == "cuisine" and  direction == "asc" %} &uarr; {% endif %}
    	    {% if sort-by == "cuisine" and  direction == "desc" %} &darr; {% endif %}
    	</a></th>
          </tr>
        </thead>
        <tbody>
          {% for food in foods %}
          <tr>
    	<td>{{food.name}}</td>
    	<td>
    	  {% ifequal food.rating 1 %}&starf;{% endifequal %}
    	  {% ifequal food.rating 2 %}&starf;&starf;{% endifequal %}
    	  {% ifequal food.rating 3 %}&starf;&starf;&starf;{% endifequal %}
    	  {% ifequal food.rating 4 %}&starf;&starf;&starf;&starf;{% endifequal %}
    	  {% ifequal food.rating 5 %}&starf;&starf;&starf;&starf;&starf;{% endifequal %}
    	</td>
    	<td>
    	  ${{food.price}}
    	</td>
    	<td>{{food.cuisine}}</td>
          </tr>
          {% endfor %}
        </tbody>
      </table>
    </div>
```

## Pagination

Now we will take a look at our pagination component. This will be placed at the bottom of the table. We will also indicate the active page with a different background highlight if the record offset values are matching with the page and the url value. We will construct the links based on the pagination
data sent by the server for the route along with other things like direction and sort-by values.

```html
    <nav aria-label="Page navigation example">
      <ul class="pagination">
        {% for page in pages %}
        <li class="page-item {% ifequal start page.start %} active {% endifequal %}" >
          <a class="page-link" href="/?start={{page.start}}&amp;direction={{direction}}&amp;sort-by={{sort-by}}">{{page.id}}</a>
        </li>
        {% endfor %}
      </ul>
    </nav>
```

## Generating pagination data

Next we are going to take a look at our utility function to generate our pagination data. We are going to use a loop with 10 iterations to create the respective pagination data for the page and the start offset value for the table data.  It will be something like for page 2, we will start with the record offset 10 and for page 3, it will be 20 and so on. Please make 
note that our records for the first page start from 0 to 9, so the second page starts from 10 and so on. 

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/fp4yjmwem65l5m73ujia.png)


We are also ensuring that the pagination data is in ascending order using the `reverse` function at the end while returning the output from the function, otherwise we will end up with pages in the descending order.
```lisp
    (defun generate-pages ()
      "Generate pagination"
      (let ((pages nil))
        (dotimes (i 10)
          (push (list :id (+ 1 i) :start (* 10 i)) pages))
        (reverse pages)))
```

# Building our data

The data for our table is just a random list of dishes, ratings, price and the cuisine. First we declare a global variable called `*foods*` and initialize the value to `nil`.
```lisp
    (defvar *foods* nil)
```

## Dishes

Next we will create a list of dish names in a separate variable called `*dishes*`.
```lisp
    (defvar *dishes* '("Pizza"
    		   "Noodles"
    		   "Fried Rice"
    		   "Roti"
    		   "Lasagna"
    		   "Churros"
    		   "Tea"
    		   "Soup"
    		   "Egg roll"
    		   "Salad"
    		   "Burger"
    		   "Rice"
    		   "Curry"
    		   "Bread"))
```

## Cuisines

Then, we will create a list of cuisine names in a variable called `*cuisines*`.
```lisp
    (defvar *cuisines* '("Indian"
    		     "Chinese"
    		     "Thai"
    		     "Continental"
    		     "Mexican"
    		     "Indonesian"
    		     "Japanese"
    		     "Spanish"
    		     "Italian"
    		     "Greek"))
```

# Generating random data

Now it's time to combine all our dish names and cuisines to generate a list of dishes with random rating values and prices. So before pushing the generated values into our global **foods** variable, let's be sure to reset the variable to nil.

Then using a `dotimes` loop for 100 iterations we are going to generate a random record for dish. We are getting a random dish and cuisine form the previously created lists called **dishes** and **cuisines** respectively. 
```lisp
    ;; Clear the list
    (setf *foods* nil)
    
    ;; Push 100 items into foods with random values
    (dotimes (i 100)
      (push (list :name (random-elt *dishes*)
    	      :cuisine (random-elt *cuisines*)
    	      :rating (+ 1 (random 5))
    	      :price (+ 1 (random 100))) *foods*))
```
For that we are using a custom defined function
called `random-elt` which will pick a random element from a list.
```lisp
    (defun random-elt (mylist)
      (elt mylist (random (length mylist))))
```
And then for the rating and price, we are using the standard library function called `random` to generate random numbers within a specified range. For example, `(random 5)` will generate random numbers between 0 and 4 and we are adding 1 to 
ensure we are getting a non-zero value.


# Sorting

Sorting data in Common Lisp is pretty easy and straight-forward when it comes to lists. We are using an higher-order function called `sort-list` which will take two parameters, the sort direction either "asc" or "desc" and the sort-by which is the key based on which we sort the list. And based on the `sort-by` key we will delegate the sorting to the respective sort functions with the direction as an argument.
```lisp
    (defun sort-list (direction sort-by)
      "Sort a list based on the direction and key"
      (cond ((string= sort-by "name") (sort-list-by-name direction))
    	((string= sort-by "rating") (sort-list-by-rating direction))
    	((string= sort-by "price") (sort-list-by-price direction))
    	((string= sort-by "cuisine") (sort-list-by-cuisine direction))))
```
Based on the direction,  we will figure out the sort function to use, `#'string>` or `#'string<` for name and cuisine, and `#'>` or `#'<` for rating and price. We can still have one function for sorting all the columns if we can refactor, because this approach will not scale for large number of columns in the table.
```lisp
    (defun sort-list-by-name (direction)
      "Sort a list by name"
      (let ((sort-fn (if (string= direction "asc") #'string< #'string>)))
        (sort (copy-list *foods*) sort-fn :key (lambda (plist) (getf plist :name)))))
    
    (defun sort-list-by-rating (direction)
      "Sort a list by rating"
      (let ((sort-fn (if (string= direction "asc") #'< #'>)))
        (sort (copy-list *foods*) sort-fn :key (lambda (plist) (getf plist :rating)))))
    
    (defun sort-list-by-price (direction)
      "Sort a list by price"
      (let ((sort-fn (if (string= direction "asc") #'< #'>)))
        (sort (copy-list *foods*) sort-fn :key (lambda (plist) (getf plist :price)))))
    
    (defun sort-list-by-cuisine (direction)
      "Sort a list by price"
      (let ((sort-fn (if (string= direction "asc") #'string< #'string>)))
        (sort (copy-list *foods*) sort-fn :key (lambda (plist) (getf plist :cuisine)))))
```

# Search route

Next we focus on the search route for our application.
The search route will take a query parameter called `query` itself, through which we will get the search keywords for the route. We will perform the search only based on the names of the dishes. We will use a utility function called `filter-foods` for this purpose.
```lisp
    (defroute ("/search" :method :POST) (&key _parsed)
      (format t "_parsed = ~a~%" _parsed)
      (let* ((query (cdr (assoc "query" _parsed :test #'string=)))
    	(filtered-foods (filter-foods query)))
        (render #P"_search.html"
    	    (list
    	     :foods filtered-foods
    	     :total (length filtered-foods)))))
```

## Filtering data

The `filter-foods` function takes the query as the parameter 
and filter out the dishes which is not matching with the name of the dish. To filter out the food list we are using the `remove-if` function with a lambda wherein we match the name of the food with the query string using the `search` function with the test as `#'char-equal`. If it matches return `NIL` so that it cannot be removed from the list , otherwise we return `T`, so that it can be removed from the list and we would only get all the matching dish names.

```lisp    
    (defun filter-foods (query)
      "Filter foods based on the query with name"
      (remove-if #'(lambda (food)
    		 (let ((name (getf food :name)))
    		   (if (search query name :test #'char-equal)
    		       nil
    		       t))) *foods*))
```

# Search template
```html
    <div id="results" >
      <p><a href="/">Clear Search</a></p>
      <p>{{total}} results found</p>
      <table class="table table-striped">
        <thead>
          <tr class="table-dark">
    	<th> <a href="/?sort-by=name&amp;direction=desc"> Name ↓</a></th>
    	<th> <a href="/?sort-by=stars&amp;direction=desc"> Stars</a></th>
    	<th> <a href="/?sort-by=price&amp;direction=desc"> Price</a></th>
    	<th> <a href="/?sort-by=category&amp;direction=desc"> Category</a></th>
          </tr>
        </thead>
        <tbody>
          {% for food in foods %}
          <tr>
    	<td>{{food.name}}</td>
    	<td>{{food.rating}}</td>
    	<td>{{food.price}}</td>
    	<td>{{food.cuisine}}</td>
          </tr>
          {% endfor %}
      </tbody></table>
    </div>
```
And this is how the final app look like in action.

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/5szvdtyjj2vfv4ds2bdy.png)


# Code

The source code for this application is hosted in [Github](https://github.com/rajasegar/cl-tabular).
If you are stuck with any step or anything is missing in this post, you can always refer to the 
updated source code in Github.