JSON validation for Cowboy REST API - Mimic Celebrate Joi in Elixir
Tags: elixir, cowboy
Published: 2022-08-15 23:30:00

When we work in a REST API one of the first things to do is validate incomming data.

Suddenly I must to rewrite a REST API from Javascript to Elixir and find that some of the existing library does not feed my basic needs of simplicity and less code possible. 
All of us who have a Node-Express background, have used Celebrate and Joi in order to validate the presence of key, their data type and their stored value.

In this article I will show you how to implement in Elixir a schema validator, trying to mimic a (minimal) functionality of Celebrate Joi. 


## Table of Contents
* [1-Plugs and Midlewares](#plugs)
* [2-Route-Schema Validation](#sample)
* [3-Function Clause Matching](#fcm)
* [4-Json Schema Validation ](#valschema)



## 1) Plugs and Midlewares <a name="plugs"></a>

There is a common architecture between Node-Express and Elixir-Cowboy, Express have middlewares and Cowboy have Plugs.
In both cases these are used for routing, body parsing, user authentication verification, cross-origin resource sharing, data logging, data validation, metrics request, etc..

How the documentation say :"An Express application is essentially a series of middleware function calls." [Using middlewars](https://expressjs.com/en/guide/using-middleware.html)

For example: A classic Node-Express midlewares chain

```javascript
//file: app_router.js
// import boiler plate ...
app.use(cors(cors_configuration)); //CORS
app.use(express.json());//Json body parsing

app.use(isAuth);//user jwt authentication verification

app.use(function (err, req, res, next) { //  error handling
  if (err.name === "UnauthorizedError") {
    res.status(401).send("invalid token...");
  } else {
    next(err);
  }
});

app.use(attachUserId);// add extra information about the current user

app.use((req, res, next) => { // some debug 
  console.log("Method",req.method);  
  console.log("Url", req.originalUrl);
  console.log("Body:",req.body);  
  return next();
});


```

The same happens in Elixir-Cowboy, where plugs are chained one after other, like this:


```elixir
# file: app_router.ex
# use/import boiler plate ...

  plug Corsica,  cors_configuration # CORS  
  plug :match  
    plug Plug.Telemetry, telemetry_configuration #  metrics request  
    # body parsing
    plug Plug.Parsers,    parsers: [:json],    pass: ["application/json"],    json_decoder: Jason

    # Json validation
    plug Plug.RouterValidatorschema, on_error: &__MODULE__.on_error_fn/2

    # user jwt authentication verification
    plug Plug.Authenticator, on_error: &__MODULE__.on_error_fn/2
    # add extra information about the current user
    plug Plug.AttachUserId, on_error: &__MODULE__.on_error_fn/2
    # some debug 
    plug Plug.PrintUrlMethod
  
  plug :dispatch


```

Celebrate is a middleware, the equivalent in Elixir is a Plug, let start writing a plug:

The next link is the started point to understand and learn about how to write a plug, I recommend read it and then come here:  [The Plug Specification](https://elixirschool.com/en/lessons/misc/plug#the-plug-specification-2)


There are two ways to define it: function plugs and module plugs, I will choose the module plugs way.


## 2) Route-Schema Validation <a name="sample"></a>

Has programers we want to write the minimal possible lines of code. Bassically reduce all to a schema definition and then apply in our route.


**Schema:**
```elixir
photo_book_schema = [
                      qty: {:number, 1, 200},
                      hardcover: {:boolean},
                      colors: {:options, ["bw", "color", "sepia"]},
                      client_name: {:string, 6, 20},
                      client_phone: {:string, 8, 25, ~r/^[\d\-()\s]+$/}
                    ]
```                        

**Route:**

```elixir
plug(:match)
plug(Plug.RouterValidatorschema, on_error: &__MODULE__.on_error_fn/2)
plug(:dispatch)

post "/create/", private: %{validate: photo_book_schema} do
  status = Orders.create(conn.body_params)
  {:ok, str} = Jason.encode(%{status: status})
  send_resp(conn |> put_resp_content_type("application/json"), str)
end

def on_error_fn(conn, errors, status \\ 422) do
  IO.inspect(errors, label: "errors")
  {:ok, str} = Jason.encode(%{status: "fail", errors: errors})
  send_resp(conn |> put_resp_content_type("application/json"), str) |> halt()
end

```

Anxious for the code ? 
Before to go deeper in the code, we learn a bite about Function Clause Matching. 


## 2) Function Clause Matching <a name="fcm"></a>

One of the powerfull tools of Elixir is pattern matching. It is applied in many places, thanks to that, we write less lines of code compared with another languages.

In this case is applied in function clauses, Elixir let us to write multiple definitions for the same function, every definitions is called a clause.

See the next example:

```elixir
defmodule Discount do
  def calculate(type, price) do
  case type do
      :normal -> price * 1
      :high   -> price * 0.8
      :low    -> price * 0.9      
  end  
  end  
end
```

Instead we can write:

```elixir
defmodule Discount do
  def calculate(:normal, price), do: price
  def calculate(:high, price), do: price * 0.8
  def calculate(:low, price), do: price  * 0.9  
end
```

Where the "case" statement is moved outside of the function and the virtual machine take the decision of which must to apply.All definition refer to the same function "calculate" with arity 2.  

### 2.1) Tips.

Some tips when use function clause matching.

#### Group:

Always group clauses of the same function together,

Don't mix with others:

```elixir

defmodule FooBar do
  def foo(:a), do: 'a'
  def foo(:b), do: 'b'
  def bar(arg), do: IO.puts arg
  def foo(:c), do: 'c'
  def foo(arg), do: IO.puts arg
end

```

Will receive a compiler warning:

```
warning: clauses with the same name and arity (number of arguments) should be grouped together, "def foo/1" was previously defined.
```
Correct way:
```elixir

defmodule FooBar do
  def foo(:a), do: 'a'
  def foo(:b), do: 'b'
  def foo(:c), do: 'c'
  def foo(arg), do: IO.puts arg
  def bar(arg), do: IO.puts arg
end

```
#### Order: 

The declaration order really matter:
Don't do:

```elixir

defmodule FooBar do
  def foo(arg), do: arg
  def foo(:b), do: 'b'  
  def foo(:c), do: 'c'
end
```

Will receive a compiler warning:

```
warning: this clause for foo/1 cannot match because a previous clause always matches
```

Correct way, sort for selective weight:

```elixir
defmodule FooBar do  
  def foo(:b), do: 'b'  
  def foo(:c), do: 'c'
  def foo(arg), do: arg
end

```

#### Match all:

Alway write a final match all clause.

Sometimes we receive a inexpectable argument value and then a error is triggered:
`(FunctionClauseError) no function clause matching`

Don't do:

```elixir

defmodule FooBar do
  def foo(:b), do: 'b'  
  def foo(:c), do: 'c'
end
```

Correct way:

```elixir

defmodule FooBar do
  def foo(:b), do: 'b'  
  def foo(:c), do: 'c'
  def foo(_), do: "Something happen"
end
```


## 4) Schema Validation <a name="valschema"></a>


The Plug.Router provide a way to [passing data between routes and plugs] 
(https://hexdocs.pm/plug/Plug.Router.html#module-passing-data-between-routes-and-plugs)

#### The [Plug.conn](https://hexdocs.pm/plug/Plug.Conn.html#module-private-fields) struct

A analogy to better undertand the `Plug.conn` and the plug chain is think in a train locomotive with wagons, when every plug is a station where we load or unload data and the wagons are the fields of the struct.

The `Plug.conn` struct have a field named `private`, it will be seted with a key and a value. 

```elixir

  post "/create/" , private: %{validate: photo_book_schema} do  
     .....
  end
```


When the plug find the `validate` key inside the `private` field, the validation process start, calling the function `validate_schema()`

```elixir

defmodule Plug.RouterValidatorSchema do

  def init(opts), do: opts

    def call(conn, opts) do
    case conn.private[:validate] do
      nil -> conn
      schema -> validate_schema(conn,schema, opts[:on_error])
    end
  end
```

The `validate_schema()` walk over the schema list and validate the keys inside `conn.body_params`
 
  If all validations are successful returns an empty map.
  Otherwise when validation fail, call the given `on_error` callback with a list of the collected errors.
  The errors list have the shape `[ %{"key1"=> "error"}, %{"key2"=> "error"}]`

```elixir

  defp validate_schema(conn, schema, on_error) do
  errors = Enum.reduce(schema, [], validate_key(conn.body_params))

  if Enum.empty?(errors) do
    conn
  else
    on_error.(conn, %{errors: errors})
  end
end

defp validate_key(data) do
  fn {key, test}, errors ->
    skey = Atom.to_string(key)
    cond do
      not Map.has_key?(data, skey) -> [%{skey => "missing key"} | errors]      
      is_nil(data[skey]) -> [%{skey => "value is nil"} | errors] # or just return errors if value can accept nil
      true ->
        case is_t(test, data[skey]) do
          :ok -> errors
          {:invalid, reason} -> [%{skey => reason} | errors]
        end
    end
  end
end

```

The `validate_key(data)` return a parameterized function that be in charge of apply every test to the key associated data.
The `is_t(test,value)` will perform the test and here is where we take advantage of function clause matching, the function test for boolean, number, string, string patterns, options, and is very easy add a new test for any desired data type.

```elixir

defp is_t({:boolean}, value) do
  if (not is_boolean(value)) do
    {:invalid, "not is boolean"}
  else
   :ok
  end
 end

 defp is_t({:number}, value) do
  if (not is_number(value)) do
    {:invalid, "not is number"}
  else
   :ok
  end
end

defp is_t({:number, vmin, vmax}, value) do
  cond do
    not is_number(value)   ->  {:invalid, "not is number"}
    value <= vmin ->  {:invalid, "min value"}
    value > vmax ->  {:invalid, "max value"}
    true ->:ok
  end
end


 defp is_t({:string}, value) do
    if is_bitstring(value) do
      :ok
    else
      {:invalid, "not is string"}
    end
 end

 defp is_t({:string, lmin, lmax}, value) do
     cond do
       not is_bitstring(value)   ->  {:invalid, "not is string"}
       String.length(value) <= lmin ->  {:invalid, "min length"}
       String.length(value) > lmax ->  {:invalid, "max length"}
       true ->:ok
     end
 end

 defp is_t({:string, lmin, lmax, reg}, value) do
  cond do
    not is_bitstring(value)   ->  {:invalid, "not is string"}
    String.length(value) <= lmin ->  {:invalid, "min length"}
    String.length(value) > lmax ->  {:invalid, "max length"}
    not Regex.match?(reg,value) -> {:invalid, "regex not match"}
    true ->:ok
  end
end

defp is_t({:options, alloweds}, value) do
  cond do
    value not in alloweds -> {:invalid, "not allowed"}
    true ->:ok
  end
end

# ---------------------------
# Final match :
# ---------------------------
defp is_t(_value,_test) do
  {:invalid, "test match fail"}
end


```