Practical OCaml
Tags: ocaml
Published: 2021-05-18 01:23:19

---
title: Practical OCaml
published: true
description: 
tags: ocaml
//cover_image: https://direct_url_to_image.jpg
---
This is a guide to OCaml as a pragmatic, general-purpose language that can scale with your everyday programming needs.

Why OCaml? I think many people have attempted to answer this question:

- https://dev.realworldocaml.org/prologue.html#scrollNav-1
- https://www2.lib.uchicago.edu/keith/ocaml-class/why.html
- https://blog.janestreet.com/why-ocaml/

But I'll try to answer from the perspective of this guide. Often when people need some automation, they will reach for Python, or nowadays maybe Go (Golang). I hope to show that OCaml can cover these needs, because it has:

- Lightweight syntax, like Python
- An interpreted mode, like Python, so you can run quick scripts immediately
- Powerful interactive environment (REPL), like Python, so you can develop and test code in an exploratory style
- Powerful type inference, so you almost never need to add type annotations to code, like Python
- Fast compiler, like Go, so you can iterate quickly
- Native compilation, like Go, so you can deploy a single binary instead of needing to set up VMs
- Efficient compiled programs, like Go, so you can enjoy instant program startup and fast runtimes

Lastly,

- A strong static typechecker (stronger than Go), so you can catch a lot of code issues very quickly.

To balance all of these out a bit, let's ask: why _not_ OCaml?

- Small, niche language and community. For certain projects that may be more critical to your company, nothing beats the ecosystem of e.g. Java, Ruby, or .NET.
- Unfamiliar syntax. This is really a sticking point for many people. I'll say a bit more about this later.

## Setup

Follow the instructions here: https://ocaml.org/install

At this point you have opam, the OCaml package manager, installed. Now, refresh the package repository info and install some standard OCaml packages:

    $ opam update
    $ opam install lwt dune utop ocaml-lsp-server alcotest odoc angstrom piaf ppx_deriving_cmdliner ppx_deriving_yojson

Explanations of these packages:

- lwt: asynchronous promises library, works very similarly to Python or Node.js async
- dune: standard build tool
- utop: REPL (interactive environment)
- ocaml-lsp-server: editor support tool
- alcotest: unit testing library
- odoc: documentation generator
- angstrom: parser combinator library, very handy for quickly parsing arbitrary data
- piaf: lightweight web client and server
- ppx_deriving_cmdliner: PPX (roughly, a macro) to generate command-line parsing code from simple OCaml types
- ppx_deriving_yojson: PPX to generate JSON encoders/decoders from OCaml types

After installing the above packages, run `opam list` to check the full list of installed packages (many were installed as dependencies of others).

Small aside: opam works by solving the dependency versions of all packages it is asked to install, then finding a set of package versions that are all consistent with each other. So ideally, we want to give it all the packages we need in one shot. Of course, we can and will install more later, but this is just good to keep in mind.

Finally, you will need to install an editor plugin to get support features like syntax highlighting, type on hover, go to definition, etc. For VS Code the recommended plugin is [OCaml Platform](https://marketplace.visualstudio.com/items?itemName=ocamllabs.ocaml-platform).

## Basic checks

Let's first do some basic checks that the system is working. Open up the REPL:

    $ utop

And enter this code at the prompt:

```ocaml
let fizzbuzz n = match n mod 3, n mod 5 with
  | 0, 0 -> "FizzBuzz"
  | 0, _ -> "Fizz"
  | _, 0 -> "Buzz"
  | _ -> string_of_int n

let () = for i = 1 to 20 do
  print_endline (fizzbuzz i)
done;;
```

Note, the `;;` is only needed to tell the REPL when the block of code is fully entered. Without it we can freely move around with arrow keys and edit the code.

The REPL should show this output:

```
1
2
Fizz
4
Buzz
Fizz
...
```

This shows the `fizzbuzz` function running for inputs from 1 to 20.

## The whirlwind tour

There is a concise tutorial at https://ocaml.org/releases/4.12/htmlman/coreexamples.html and it's a really good idea to go through it to get the basics. But a super concise crash course:

- Bind a variable to a value:

```ocaml
let x = 1
```

- Create a function:

```ocaml
let add_one x = x + 1
```

The `x` is the parameter and the expression on the right of the `=` sign is the function body. A good way to think about OCaml functions is the substitution model: functions are evaluated by substituting their parameters with the arguments that are actually given when the function is called. So e.g.

```ocaml
add_one x = x + 1
add_one 0 = 0 + 1
          = 1
```

Technically, bindings have _patterns_ on the left-hand side of the `=` sign. Patterns are any _literal_ OCaml value or structure, or variables, or some combination of the two. E.g., the following are valid bindings:

```ocaml
let 1 = 1
let 2 = 1
```

Of course, we don't normally bind like this because it can (and usually will) throw an exception at runtime, when OCaml finds out that the values are not actually the same. But it's good to keep this piece of knowledge in the back of your mind for later.

- Define a variable in a limited scope:

```ocaml
let print_sum x y =
  let sum = x + y in
  Print.printf "Sum: %d\n" sum
```

The `let ... in ...` syntactic form is composeable, so we can nest them:

```ocaml
let print_sum x y =
  let sum = x + y in
  let sum_string = string_of_int sum in
  Printf.printf "Sum: %s\n" sum_string
```

It may be easier to understand the structure if we indent it like this:

```ocaml
let print_sum x y =
  let
    sum = x + y
  in
    let
      sum_string = string_of_int sum
    in
      Printf.printf "Sum: %s\n" sum_string
```

But unlike Python, OCaml syntax is not whitespace-sensitive, and is usually written in a compact style.

- Define a record type (like a Python namedtuple or a Go struct):

```ocaml
type file = { name : string; contents : string }

let my_file = { name = "README.md"; contents = "Hello!" }
```

Note: in OCaml, strings are just bytestrings. They aren't assumed to have any encoding e.g. UTF-8. If we need Unicode support we can use libraries like Camomile to get it. But often, we don't need it because we're just shuttling strings back and forth or handling them in a very limited way.

- Define a variant type (like an enum type but more powerful):

```ocaml
type payment_method =
| Cheque of string
| Credit_card of string * string * string

let to_string payment_method = match payment_method with
  | Cheque number ->
    Printf.sprintf "Cheque # %s" number
  | Credit_card (name, number, expiry) ->
    Printf.sprintf
      "Credit card # %s, cardholder name %s, expiry %s"
      number
      name
      expiry
```

Note: `match ... with ...` is a super important construct in OCaml. It is the main workhorse for logic algorithms and ensuring all cases are covered. Here's an example of what it can do: https://www.reddit.com/r/programming/comments/n2639k/ocaml_typechecker_catches_a_redundant_rule_in/

In this example `Cheque` and `Credit_card` are called 'cases' or 'constructors'. If you've ever done some high school algebra you'll have come across functions that are defined by case analysis, e.g.:

    f(0) = 0
    f(x) = 1/x

Variants and pattern matching are a way of bringing this notation into programming.

- Define and use modules:

In OCaml, _modules_ are the unit of organization of code. They also serve as the unit of compilation, and as namespaces (among other uses). Modules are very powerful and are one of the 'secret weapons' in OCaml.

In an OCaml project, each source file automatically becomes a module. E.g. if you have a file `myprog.ml`, the compiler derives a module `Myprog` from it (the first letter is capitalized). All modules in a project are automatically 'in scope', or visible. So we don't need to import anything. E.g. if you have a source file:

```ocaml
(* lib.ml *)
let x = 1
```

Then the module `Lib` is available to the rest of your program, and so is the value `Lib.x`. This is one of the things that makes programming in OCaml so productive–the compiler takes care of mundane details like importing values.

- Define nested modules:

```ocaml
(* lib.ml *)

module Print_endline = struct
  let int = Printf.printf "%d\n"
  let string = print_endline
end

let print_person name age =
  Print_endline.string name;
  Print_endline.int age
```

This is one of the killer features of modules that few other languages possess. You can tie together discrete units of code inside files and use them to neatly organize, document, and control visibility of the code.

Note: in OCaml, we often want to perform some operations (let's informally call them 'actions') in sequence. Actions are just expressions that return no meaningful value (in other words, `()`), and are composed using the `;` operator, so e.g. `ACTION1; ACTION2; ...; ACTIONn`. We typically say that actions are performed solely for their _side effects_ e.g. printing something out.

- Run the 'main' program:

```ocaml
let () = print_sum 1 2
```

Unlike Go or Python, there is no specific main entrypoint into a program in OCaml. Any code that is not a function definition is executed immediately when a module is loaded during program startup.

Typically, we arrange our programs so that there is one obvious entrypoint module, perhaps named `main.ml`, and bind the main code (actions) to a `()` pattern to indicate that it returns no meaningful result. This is a safe binding because the `()` value (pronounced 'unit') is the _only_ value of its type, `unit`. So there's no possible way for it to fail to match or throw an exception at runtime.

Of course, if you are using it for scripting, then you would call the file whatever is appropriate and put the entrypoint in there.

## Proof of concept

Let's make a small project in OCaml to prove out its practicality for general-purpose programming. We're going to implement a StatsD filter proxy like the one described by Alan Ning in [Optimizing 700 CPUs Away With Rust](https://medium.com/tenable-techblog/optimizing-700-cpus-away-with-rust-dc7a000dbdb2). Be prepared to read and try to get your head around some OCaml code here (not a lot though)!

Note, this is not about competing with Rust–benchmarks are not my goal here–but about proving out OCaml's feasibility for all kinds of projects.

Short recap: StatsD is an application performance monitoring tool that runs as a daemon. Any application can send statistics to it. The statistics are simple lines that look like:

```
foo:1|c
```

That is a metric named `foo`, which is a counter (`c`), and here we are incrementing the counter by `1`. (As an aside, I would have chosen the format `foo:c=1`, but that seems obvious to me as an OCaml programmer!)

The protocol is super simple–not even HTTP, just raw UDP datagrams.

The filter proxy in this small project is a service that receives these metrics lines, filters them out by checking against a blocklist, and forwards any unfiltered metrics lines to the actual StatsD daemon.

First, create a directory `ocaml_statsd_filter` to house the project:

    $ mkdir ocaml_statsd_filter
    $ cd ocaml_statsd_filter

Now, create the project files:

- `dune-project`:

```
(lang dune 3.6)
```

- `dune`:

```
(executable
  (name ocaml_statsd_filter)
  (libraries str unix))
```

This file describes the main executable, named `ocaml_statsd_filter`, and what _libraries_ it depends on. Libraries are basically groups of (one or more) modules that are distributed together. The two libraries used here are `str` for regular expressions and `unix` for Unix programming (communicating with sockets, managing files, etc.). These two happen to be distributed with OCaml but need to be explicitly listed as dependencies because they're not portable to all platforms.

To learn more about dune, visit its website: https://dune.build/ .

- `cfg.ml`:

```ocaml
let listen_port =
  try int_of_string (Sys.getenv "listen_port") with Not_found -> 8125

let forward_host = try Sys.getenv "forward_host" with Not_found -> "127.0.0.1"

let forward_port =
  try int_of_string (Sys.getenv "forward_port") with Not_found -> 8126

let blocklist =
  try
    "blocklist"
    |> Sys.getenv
    |> String.split_on_char ',' 
    |> List.map Str.regexp_string 
  with
    Not_found -> []
```

This helper module reads configuration values that are passed in as environment variables:

  - `listen_port`: what port to listen on for metrics
  - `forward_host`: what host to forward the metrics to
  - `forward_port`: what port to forward the metrics to on `forward_host`
  - `blocklist`: comma-separated list of words to block, e.g. `foo,bar`, split into a list of regular expressions that we can check against

If any of the above environment variables are not found, it uses OCaml's convenient exception handling to substitute defaults.

Getting configuration from environment variables is as simple as it gets! We could easily have done something more complex though, like reading a JSON file to get the configuration. In practice, environment variables are often more than good enough, and easy to feed into services using tools like [systemd](https://www.freedesktop.org/software/systemd/man/systemd.exec.html#Environment=).

Note, the `|>` operator is called 'pipe-forward', and it lets you write a series of function applications in a left-to-right direction without parentheses, instead of right-to-left. [This TC39 proposal](https://github.com/tc39/proposal-pipeline-operator/#introduction) to add it to JavaScript explains the reasoning fairly well. (It comes out of the box with OCaml.)

- `ocaml_statsd_filter.ml`:

Before we get to the code, just a note that the name of this file is significant–we need a file name that matches the name given in the `name` field in the `dune` file. In other words, we have `(name ocaml_statsd_filter)` in the `dune` file, so we need to have a module `Ocaml_statsd_filter` in the project.

```ocaml
open Unix

let bufsize = 8192
let buf = Bytes.create bufsize

let forward_addr = ADDR_INET (inet_addr_of_string Cfg.forward_host, Cfg.forward_port)
let forward_sock = socket PF_INET SOCK_DGRAM 0

let allow data = Cfg.blocklist
  |> List.exists (fun regexp -> Str.string_match regexp data 0)
  |> not

let process inc _ =
  let in_descr = descr_of_in_channel inc in
  let read_len, _ = recvfrom in_descr buf 0 bufsize [] in
  let buf_str = Bytes.to_string buf in
  if allow buf_str then begin
    ignore (send forward_sock buf 0 read_len []);
    print_string ("Sent: " ^ buf_str)
  end else
    print_string ("Did not send: " ^ buf_str)

let () =
  connect forward_sock forward_addr;
  establish_server process (ADDR_INET (inet_addr_any, Cfg.listen_port))
```

This is the main executable. It works like this:

- For each incoming request, fork a new process
- In the process to handle the request, read 8192 bytes (i.e., 8 KB) of data from the request, and forward it to the forwarding address _if_ it doesn't contain any blocklisted words

This is all from built-in functionality in the `Unix` module:

- Multi-process request handling server
- Using sockets to communicate

We are using the `Unix` functionality so heavily in this program that we just `open` the module at the top of the file, something that we rarely do in OCaml programs, because `open`ing modules liberally pollutes the current scope with the contents of those modules and makes code harder to understand.

In this case it's worth it because it's a small module and almost all the functionality in it is from `Unix`. You can hover over the various functions in your editor to get the types and documentation about them. Full documentation is also available at https://ocaml.org/releases/4.12/api/Unix.html .

By the way, the name `Unix` is actually a slight misnomer; the functionality is _mostly_ portable to Windows–exceptions are noted in the documentation. The idea behind the module is to provide a common set of system programming functionality, very similar to C.

A few points to note about this server:

- It first sets up a socket pointing to the 'upstream' server (the one that it forwards the metrics to), since it will continuously send data to it
- It doesn't establish a connect to the 'downstream' clients; it just reads data from them in unconnected mode
- It reads only 8 KB of data by default; this should be enough for StatsD metrics lines based on the [StatsD documentation](https://github.com/statsd/statsd/blob/9cf77d87855bcb69a8663135f59aa23825db9797/docs/metric_types.md#multi-metric-packets). Also, that's what Alan Ning's [Rust implementation](https://github.com/tenable/statsd-filter-proxy-rs/blob/6ba64dabc280b3e483a93f29df93688945b7f8fc/src/server.rs#L17) does.

For more Unix socket programming in OCaml, check [this chapter](https://ocaml.github.io/ocamlunix/sockets.html) of the OCaml Unix book.

## Test run

Now let's test out the PoC:

    $ OCAMLRUNPARAM=b blocklist=foo,bar dune exec ./ocaml_statsd_filter.exe

The main command here is `dune exec ocaml_statsd_filter.exe`. The `.exe` extension tells dune to build and run a binary executable. Building a bytecode version is also possible, but in this guide we will test the binary version only.

The two environment variables passed to the process are:

- `OCAMLRUNPARAM=b`: this tells OCaml to print a full stack trace if there is an exception; normally it only prints the exception message
- `blocklist=foo,bar`: this is handled specifically by our `Cfg` module above, and blocks any metrics with the words `foo` or `bar` from being forwarded upstream

In a separate terminal window, try sending messages which should be blocked:

    $ echo 'foo:1|c' | nc localhost 8125
    $ echo 'bar:1|c' | nc localhost 8125

Then check the server–it should show the following messages:

    Did not send: foo:1|c
    Did not send: bar:1|c

Next, try sending a message which _shouldn't_ be blocked:

    $ echo 'baz:1|c' | nc localhost 8125
    $ echo 'baz:1|c' | nc localhost 8125

(We need to do this a couple of times to trigger the error we want to see on some systems, because of [a peculiarity](https://stackoverflow.com/a/41231889/20371) in how UDP connections work.

Then check the server:

    Fatal error: exception Unix.Unix_error(Unix.ECONNREFUSED, "send",
    "")
    Raised by primitive operation at file "unix.ml", line 642,
    characters 7-39
    Called from file "ocaml_statsd_filter.ml", line 17, characters 11-48
    Called from file "unix.ml", line 1195, characters 12-37
    Called from file "ocaml_statsd_filter.ml", line 24, characters 2-71

This throws an exception because there is no server upstream listening to the port we're sending to right now. If you do happen to have a StatsD daemon running on the port, it should work!

## Review

I hope this gives you an idea that OCaml is surprisingly capable out of the box. A few key points to note:

- There really are no type annotations needed anywhere. It looks just like a scripting language. People sometimes don't like this and want to annotate everything. My approach is to let my editor plugin and my compiler tell me what the types are and if I'm getting it wrong–it's easy to fix. And as you work in the codebase over time, you get used to not having any explicit types in the implementations, just like dynamic typing programmers do, but with the added benefit of static typing!
- The syntax is a bit weird though. There is a reason why each piece of the syntax is the way it is–it's a balance between accreting new language features over time and preserving a fast, unambiguous syntax for a fast parser. But for those who can't get used to it, [ReasonML](https://reasonml.github.io/) syntax is always a possibility–it's easy to install with opam (`opam install reason`), and dune supports it out of the box (just start writing `.re` files instead of `.ml`, and so on). It's a familiar syntax that's designed to look like JavaScript but compile to the same native executable that regular old OCaml syntax does.
- We didn't actually end up using any of the opam libraries we installed earlier. But, stay tuned!