CSV parsing with Scala and shapeless
Tags: akkastreams, shapeless, scala, programming
Published: 2020-05-08 08:38:03

---
title: CSV parsing with Scala and shapeless
published: true
date: 2018-12-17 08:14:48 UTC
tags: akkastreams,shapeless,scala,programming
canonical_url: 
---

#### The [shapeless](https://github.com/milessabin/shapeless) library serves as an excellent foundation for building generic, reusable components. We demonstrate using the types `HList` and `Generic` to parse strings into case classes.

### Introduction

This post complements the upcoming article [Real-time log processing with Akka streams](https://www.becompany.ch/en/blog/2016/07/29/realtime-log-processing), which involves processing log entries from a web server log. For convenience and clarity, we want to parse the log file line strings into case class instances, allowing us to process them in a semantic fashion.

There are countless libraries available for parsing strings; we will use [scala-csv](https://github.com/tototoshi/scala-csv) and [shapeless](https://github.com/milessabin/shapeless) to demonstrate a generic and extensible approach. Some of the code we use is based on the [CSV example](https://github.com/milessabin/shapeless/blob/master/examples/src/main/scala/shapeless/examples/csv.scala) in the shapeless codebase.

### Get the source

The source code of the example project is [available on github](https://github.com/becompany/akka-streams-example).

### Log format

Our example uses a web server log format with the following components:

1. Remote IP address
2. Time in milliseconds (Unix time stamp)
3. Request path
4. User agent

This roughly corresponds to the following nginx log configuration:

```
log\_format my\_log\_format
  '"$remote\_addr","$msec","$request","$http\_user\_agent"';
```

A log entry looks like this:

```
"1.2.3.4","1466585706027","/foo","Chrome"
```

We will use the following case class to represent log events:

{% gist https://gist.github.com/devkat/a816ce83ff785dd594259b57d814dea6 %}

### Parsing string values into objects

A parser type for individual CSV record elements is defined by the [`Parser`](https://github.com/becompany/akka-streams-example/blob/master/src/main/scala/ch/becompany/akka/io/csv/Parser.scala) trait. The parse result is either an object of type `T` or an error message:

{% gist https://gist.github.com/devkat/abb4314649fa55386134b744b7fb333a %}

For convenience, we provide a parser which handles exceptions and returns an error message. This covers the typical scenario of using parser methods from Java libraries, e.g. `Integer.parseInt`, which throws `NumberFormatException`, or `DateFormat.parse`, which throws `ParseException`:

{% gist https://gist.github.com/devkat/cfa530cd0cd1ddf8e83f23aaac234ad3 %}

The [`Parsers`](https://github.com/becompany/akka-streams-example/blob/master/src/main/scala/ch/becompany/akka/io/csv/Parser.scala) object provides a trivial parser for strings. The client application can provide additional custom parsers. Our application provides a parser for parsing millisecond expressions into Joda Time `Instant` objects and a parser for IP addresses:

{% gist https://gist.github.com/devkat/9b795674aed1ef784863e81c39d809d9 %}

### Parsing CSV records

The [`LineParser[T]`](https://github.com/becompany/akka-streams-example/blob/master/src/main/scala/ch/becompany/akka/io/csv/LineParser.scala) class parses a CSV record, which is represented as a `List[String]`, into an instance of case class `T`. We use the `HList` (heterogenous list) type from the shapeless library to express the types of the list elements. The string list is first converted to a `HList` instance, which in turn is converted to a case class instance.

We will need some types from the shapeless library:

{% gist https://gist.github.com/devkat/ea14e78c88ad075a1cfa5067add342a5 %}

First we define a `LineParser[Out]` trait which defines the capability of parsing a list of strings into an object of type Out. In the case of an error, we expect that the parsing errors of individual list elements will be aggregated into a list represented by the Left incarnation of the `Either` result type.

{% gist https://gist.github.com/devkat/3a209c2f12ecb487e3416698f9c18fcd %}

Now we define a companion object for our `LineParser` trait which provides methods for parsing `List[String]` instances into `HList` instances. The methods are declared as implicit, making them available in the `LineParser `trait. The object provides a method for each of the `List` incarnations `Nil` and `Cons` (`::`, although we are using `+:` due to a name collision with the shapeless `::` type).

{% gist https://gist.github.com/devkat/6103f5f0929042e54d32e178e3480b23 %}

The `hnilParser` method is expected to emit an empty list and returns an error if it encounters a list containing one or multiple elements:

{% gist https://gist.github.com/devkat/beab15a89a12db73bed9d9a2fe981b7e %}

The `hconsParser` method parses a concatenation of a head element (type `H`) and a tail list (type `T`) while combining the errors which may occur during the parsing steps accordingly. Note the `Parser` type tag for the `H` type which ensures that the list element type `H` is a member of the `Parser` type class, i.e. that a parser for this type is provided. The parser is later on obtained using the expression `implicitly[Parser[H]]`.

{% gist https://gist.github.com/devkat/cf7c27a26bb5126659066b6f502989cc %}

The following implicit method converts the `HList` `R` into the case class `A`, using the shapeless `Generic` type. We define an implicit parameter gen, whose type is an instance of the `Generic` trait with a representation of type `R`. The method call `gen.from` converts the `HList` representation into the desired instance of case class `R`.

{% gist https://gist.github.com/devkat/505246aae11b585d955922d3b222001c %}

The `apply[A]` method is parameterised with the type of the expected case class. It uses the implicitly provided parser to parse the list elements. The implicitly available method `caseClassParser` allows us to use a case class as the type parameter for the apply method.

{% gist https://gist.github.com/devkat/03d35c02958de35580f5df7437dec73b %}

### Putting it all together

The [`CsvReader[T]`](https://github.com/becompany/akka-streams-example/blob/master/src/main/scala/ch/becompany/akka/io/csv/Csv.scala) class converts comma-separated strings into objects of type `T`.

The read method parses the lines of an Akka stream source, which emits elements of type String, into objects of type `T`. Note that the type parameter `T` has a context bound of type `LineParser`, which ensures that a parser is available for this type, as explained in the previous section.

The class uses the CSV parser class from the scala-csv library to split lines into lists of strings. In case the line was successfully split into a list of strings, a `LineParser[T]` is created and its apply method is invoked on the string list.

{% gist https://gist.github.com/devkat/0e15fec79fe27b9da5713324d0bd3fe6 %}

The `CsvReader[T].read` method can now be used to transform a source of strings into a source of elements of type `T`:

{% gist https://gist.github.com/devkat/4a43771642c7461f2a90b2a48a8fe6b3 %}

### Further reading

- [Type classes and generic derivation](https://meta.plasm.us/posts/2015/11/08/type-classes-and-generic-derivation/) on meta.plasm.us
- [shapeless](https://github.com/milessabin/shapeless) on github
- [scala-csv](https://github.com/tototoshi/scala-csv) on github

_This article was originally published on the_ [_BeCompany blog_](https://www.becompany.ch/en/blog/2016/07/13/csv-parsing)_._

* * *