Write a simple Web Framework in Nim language from scratch
Tags: web, nim, framework, python
Published: 2020-04-24 06:42:59

## What is “from scratch”?

Sometimes we are told “Don’t reinvent wheel”. It may be right when we want to write reliable and stable software. We should rely on mature framework instead of write our own framework. However sometimes life is too boring if we don’t challenge ourselves. If we want to learn how web frame works, we should write it by ourselves. “I see and I remember. I do and I understand.”

Ok, you maybe want to write your own web framework “from scratch”. Well, now you have two problems :-). What is “from scratch” ? Does it mean that we need to reinvent “semiconductor”. Of course not. According to what you want to learn, you can “stand on the shoulders of giants”. If you want to learn low level module, you can write it beginning from TCP or HTTP Protocol. And you can write a HTTP server and make a web framework based on it.

Regarding this article, we will rely on others’ HTTP server, because I just want to know how a web framework works.

## What is web framework?

Web framework is also called web application framework. It is built on the HTTP server which is built on HTTP protocol.  Generally, HTTP server is only used to accept request from client and respond to client. Web framework supplies more useful and powerful utilities such as data validation or user authentication. 

For example, when you shop online, you click on purchase button. Your browser will send a request to web server. Web server will parse request into data structure which web framework can understand. Web framework will verify your identity, add this product or user info into database and send a request to deduct money from your credit card and so on.

## What we need mostly?

[Nim](https://nim-lang.org/) programming language

Standard libraries: `asyncdispatch` and `asynchttpserver `.

## Why Nim?

[Nim](https://nim-lang.org/) is a statically typed compiled systems programming language. It has intuitive and clean syntax. Nim is efficient, expressive and elegant. Choose [Nim](https://nim-lang.org/) and enjoy your life!

I love it because of three reasons:

- Indent and elegant syntax
- Static Type
- High performance

## Simple usage of asynchttpserver

Nim has a built-in asynchronous HTTP server namely `asynchttpserver`. You don’t need to install it, just type `import asynchttpserver` to use it. 

HTTP server helps us transfer contents(like HTML, Json and Text) to our client(like browser or curl).  It will parse request into `seq` or `tables` in Nim language which our web framework can understand.

Let’s look at the code.

`Http200` is a status code which tells client that everything is ok. “Hello World” is the text we responds to the client with. We also need to respond to the client with HTTP headers. 

Now run the code below and enter `localhost:8080` in your browser, you will get `Hello World` on the screen.

```nim
# nim c -r thisfile.nim
import asynchttpserver, asyncdispatch

var server = newAsyncHttpServer()
proc cb(req: Request) {.async.} =
  await req.respond(Http200, "Hello World", newHttpHeaders())

waitFor server.serve(Port(8080), cb)
```

Let’s look at request headers and response headers to get more intuitions.

**request headers**

```
GET / HTTP/1.1       # start line
Host: 127.0.0.1:8080 # request headers
......
```

**response headers**

```nim
HTTP/1.1 200 OK      # start line
Content-Length: 11   # response headers
```

Now let’s extend out framework!

## Serve static files

Sometimes we want to serve static files like CSS, HTML, Images and so on. It is wise to use Nginx. But for small applications, we just want to use ourselves’ framework.

In Nim, you only need to prepare strings and send strings to client. If you want to send HTML files, you read it and send it by socket. If you want to send Json text, first convert/dump it to strings and then send it by socket.

Now follow three steps:

Firstly, judge whether the file exists. Secondly judge whether we have access to files. Finally judge whether we can open files.

Let’s look at **Pseudo code**. `resp` followed by contents we give to `respond` proc in `asynchttpserver`.

**note: await must be used in functions**

```nim
import asyncdispatch, asynchttpserver, os


proc serveStaticFile*(req: Request, dir, filename: string) {.async.} =
  # exist -> have access -> can open
  let path = dir / filename

  # whether exists file
  if not existsFile(path):
    await req.respond(Http404, "File doesn't exist.", newHttpHeaders())

  # whether has access to file
  var filePermission = getFilePermissions(path)
  if fpOthersRead notin filePermission:
    await req.respond(Http403, "You have no access to the file.", newHttpHeaders())

  # whether can open file
  try:
    let content = readFile(path)
    await req.respond(Http200, content, newHttpHeaders())
  except IOError:
    await req.respond(Http404, "404 Not Found.", newHttpHeaders())
```

## Basic Route

Route is used to map URL to corresponding proc which responds to client with contents. 

**For example**:

This example demonstrates that web framework execute some actions such as fetch info from database,  login, return HTML and so on according to corresponding HTTP method and URL. 

- get 127.0.0.1:8080/hello -> send email
- get 127.0.0.1:8080/home -> fetch info from Database

```nim
import httpcore


proc findHandler*(httpMethod: HttpMethod, path: string) =
  case httpMethod
  of HttpGet:
    case path
    of "/hello":
      echo "get 127.0.0.1:8080/hello"
      # sendEmail()
    of "/home":
      echo "get 127.0.0.1:8080/home"
      # fetchInfoFromDatabase()
    else:
      discard
  of HttpPost:
    case path
    of "/hello":
      echo "post 127.0.0.1:8080/hello"
      # login()
    of "/home":
      echo "post 127.0.0.1:8080/home"
      # returnHome()
    else:
      discard
  else:
    discard
```

### Use Hash Map for static route

If client requests `www.example.com/login`, our application will look up hash table to find corresponding Handler. Handler is the proc which handles request from client and generates response.

It is fast to look up URL in hash table.

```nim
type
  Router* = Table[URL, procHandler]
```

### Dynamic route

Sometimes we need dynamic URL to meet our demands. For example, we use only one `procHandler` to handle login operations for different customers. So every customer will have different login URLs like `www.example.com/login/1`, `www.example.com/login/2` and so on.  

We need URL match like `www.example.com/login/{id}` to catch different ids.

First we split URL with `/` , turn `www.example.com/login/1` into `@[“www.example.com”, “login”, “1”]` . Then we iterate hash table to decide whether URL match patterns.

Let’s look at **Pseudo code**.

```nim
let routeList = route.split("/")
# iterate all URL and procHandler pairs
let pathList = iterate(HandlerTable)
for idx in 0 ..< pathList.len:
  # if match continue 
  # www.example.com => www.example.com
  # login => login
  if pathList[idx] == routeList[idx]:
    continue

  # match {id} => 2
  if routeList[idx].startsWith("{"):
    let key = routeList[idx]
    if key.len <= 2:
      raise newException(RouteError, "{} shouldn't be empty!")
    let
      params = key[1 ..< ^1]
```

### Regex Route

We can use `seq` to store regex route. 

`/post(?P<num>[\d]+)` => `/post521` or `/post1314` and gets matched parameters such as `num = 521` or `num = 1314`. 

```nim
type
  ReRouter* = seq[(URL, ProcHandler)]
```

We just iterate URL and procHandler in Reroute to match request URL. 

Let’s look at **Pseudo code**.

```nim
# find regex route
for (URL, ProcHandler) in reRouter:
  if path.httpMethod != URL.httpMethod:
    continue
  var m: RegexMatch

  # save matched params like id = 2
  if URL.route.match(path.route, m):
    for name in m.groupNames():
      pathParams[name] = m.groupFirstCapture(name, URL.route)
```

## Cookie

HTTP protocol is stateless. But we use plain Cookie to save user information.

Cookie is HTTP headers. It can carry user insensitive information. When you want to carry sensitive information, make sure encrypt it using Encryption Algorithm such as Sha-256, Sha-512 and so on. Only use Cookie with HTTPS to prevent it from man-in-the-middle attacks.

This is what cookie looks like. It is in the form of `name-value` pair. Different pairs are separated by semicolon.

```html
Cookie: username=flywind; age=21
```
**Parse CookieJar**

We use `StringTable` to store different `name-value` pairs. 

```nim
type
  CookieJar* = object
    data: StringTableRef

proc parse*(cookieJar: var CookieJar, text: string) {.inline.} =
  var 
    pos = 0
    name, value: string
  while true:
    pos += skipWhile(text, {' ', '\t'}, pos)
    # name = username
    pos += parseUntil(text, name, '=', pos)
    if pos >= text.len:
      break
    inc(pos) # skip '='
    # value = flywind
    pos += parseUntil(text, value, ';', pos)
    # username = flywind
    cookieJar[name] = move(value)
    if pos >= text.len:
      break
    inc(pos) # skip ';'
```

Also we want to set cookie.

```nim
import options, times, strtabs, parseutils, strutils


type
  SameSite* {.pure.} = enum
    None, Lax, Strict

  Cookie* = object
    name*, value*: string # root => admin
    expires*: string
    maxAge*: Option[int]
    domain*: string
    path*: string
    secure*: bool
    httpOnly*: bool
    sameSite*: SameSite
```

**Set cookie**

```nim
proc setCookie*(cookie: Cookie): string =
  result.add cookie.name & "=" & cookie.value
  if cookie.domain.strip.len != 0:
    result.add("; Domain=" & cookie.domain)
  if cookie.path.strip.len != 0:
    result.add("; Path=" & cookie.path)
  if cookie.maxAge.isSome:
    result.add("; Max-Age=" & $cookie.maxAge.get())
  if cookie.expires.strip.len != 0:
    result.add("; Expires=" & cookie.expires)
  if cookie.secure:
    result.add("; Secure")
  if cookie.httpOnly:
    result.add("; HttpOnly")
  if cookie.sameSite != None:
    result.add("; SameSite=" & $cookie.sameSite)
```


## Middleware

Middleware is used to execute some operations before procHandler and after procHandler. For example, we want to verify user identity, we can use middleware to do this work.

Let’s look at **Pseudo code**.

```nim
proc verify(user: User, next: Handler) =
  # check whether user is valid
  if not user.isValid:
  	resp 404, "You have no access to this URL."
  	return
  
  # call next middleware or procHandler
  next()
  
  # If user is valid, print "Welcome" message
  resp "Welcome, " & $user 
```

There are two ways to implement  middleware, one is based on function call, the other is based on hook.

### Let’s look at hook way

We set `before` and `after` function type for every Middleware object. 

```nim
type
  Middleware = object
  	before: proc()
  	after: proc()
```

In the lifetime of application, assuming that we have two middlewares. So the order of execution is: 

m1.before -> m2.before -> hello -> m2.after -> m1.after

Let’s look at **Pseudo code**.

```nim
proc hello() =
  resp "Hello"
  
var app = Application()
var m1 = Middleware(...)
var m2 = Middleware(...)
app.addRoute(procHandler = hello, middlwares = [m1, m2])
```

### Let’s look at Function call way

We use `seq` to store middlewares.  `await next` is used to call next middleware or procHandler.

We have `size` variable to get current middleware. In the beginning, `size` is 0 and we get first middleware. Now we begin to execute the first middleware. When we encounter `await next`. `size` will increase by one, and we will call the second middleware. Finally we call the last middleware, and execute the rest program.

m1 -> m1.next -> m2 -> m2.next -> hello -> m2 -> m1

Let’s look at **Pseudo code**.

```nim
type
  Middlewares* = seq[procHandler]
  
proc httpRedirectMiddleWare*() =
  case request.scheme
  of "http":
    setScheme(request, "https")
  of "ws":
    setScheme(request, "wss")
  else:
    return
    
  # Will call next middleware or procHandler
  await next()
  
  response.code = Http307
```

## Exception Handler

We can map HTTP status codes to user-defined exception Handler.

Sometimes, we want to associate HTTP status code to unified pages for example custom 404 pages.

```nim
type
  ErrorHandlerTable* = Table[HttpCode, ErrorHandler]

proc default404Handler*() =
  response.body = errorPage("404 Not Found!", PrologueVersion)
  
app.errorHandlerTable.add(Http404, default404Handler)
```

Let’s look at **Pseudo code**.

```nim
if response.code in app.errorHandlerTable:
  await (app.errorHandlerTable[response.code])()
```

## More part

Regarding web framework, there are more parts than what I mentioned, you can checkout 

- Web framework [Prologue](https://github.com/planety/prologue) 

- Network infrastructure [Netkit](https://github.com/iocrate/netkit)