Getting started with Apache Beam and Kotlin
Tags: programming, apachebeam, streaming
Published: 2020-04-08 19:17:13

---
title: Getting started with Apache Beam and Kotlin
published: true
date: 2020-04-08 18:48:35 UTC
tags: Programming,Apache Beam,streaming
canonical_url: https://www.codingblocks.net/programming/getting-started-with-apache-beam-and-kotlin/
---

Are you looking for a simple getting started guide for working with Kotlin and Apache Beam? Well congratulations, you are in the right place. I’ve been doing some [live coding on the subject](https://www.youtube.com/watch?v=Ycim03b24CU&list=PLWWyzc5ehM91WKA467swlSgR1r_2m5LMJ), and I made a teeny tiny project to help me get started on new Beam projects quicker.

There is an [official guide](https://beam.apache.org/get-started/wordcount-example/#minimalwordcount-example) for getting started, so you should probably start there. However, that guide begins with a fully functional project and I prefer to build a minimum viable product from scratch that is easy to run in an IDE. So if you are like me, and prefer to type your own code then keep reading!

Note: If you don’t like typing either, you can just clone the [repo](https://github.com/codingblocks/beam-kotlin-example) and you are ready to go

## Wait, what is Beam?

[Beam](https://beam.apache.org/) is an open-source and unified model for interacting with streaming and batch data. Have you heard of projects or methodologies like [Spark](https://spark.apache.org/), [Flink](https://flink.apache.org/), [MapReduce](https://en.wikipedia.org/wiki/MapReduce) or batch processing? If not, then this post will not be useful for you. If you have, though, and you haven't heard of Beam - then please let me introduce you! 

Beam is a project that lets you choose a supported language (currently Java, Python, and Go), and write code can run in either batch or streaming mode, and can run in a variety of engines (Spark, Flink, Apex, Dataflow, etc). That means you get to focus on your business logic, and let Beam worry about the implementation. Check out the [compatibility matrices](https://beam.apache.org/documentation/runners/capability-matrix/) for more information.

Is this the future of batch and streaming data?
Maybe! Time will tell but I think it's worth investigating if these areas interest you.

## Create the project

First up, “File” > “New Project” and choose “Maven”. Check the “Create from archetype” checkbox and select the **org.jetbrains.kotlin:kotlin-archetype-jvm** , because Kotlin is awesome.

![](https://www.codingblocks.net/wp-content/uploads/2020/04/image-2.png)<figcaption>Choose the Kotlin JVM archetype, but you can use the default settings for everything else.</figcaption>

## Add the minimal dependencies

I’m aiming for a bare-bones implementation here, so I’m just going to add the dependency for the direct runner directly to my project for now. Later we can break this out to a “profile” so you can optionally compile in the bits for the execution engines of your choice.

```xml
<!-- Add these to your dependencies in pom.xml -->
<dependency>
  <groupId>org.apache.beam</groupId>
  <artifactId>beam-sdks-java-core</artifactId>
  <version>2.19.0</version>
</dependency>

<!-- This dependency is for the local runner. -->
<!-- Later we can move this one out to a profile -->
<dependency>
  <groupId>org.apache.beam</groupId>
  <artifactId>beam-runners-direct-java</artifactId>
  <version>2.19.0</version>
  <scope>runtime</scope>
</dependency>
```

## Create a simple pipeline

Now let’s setup a simple pipeline that is essentially a stripped down version of the [official example](https://beam.apache.org/get-started/wordcount-example/#minimalwordcount-example).

I’ll paste the code below, but here is the outline of what is happening:

1. Create a pipeline with default options
2. Add a step to read from pom.xml
3. Add a step to split the text into text characters
4. Add a step to count the characters
5. Add a step to format the output to a collection of strings
6. Add a step to write the output to files, prefixed with “counts:
7. Execute the pipeline

```kotlin
fun main(args: Array<String>) {
    val p = Pipeline.create()
    p
            .apply<PCollection<String>>(TextIO.read().from("pom.xml"))
            .apply(
                    FlatMapElements.into(TypeDescriptors.strings())
                            .via(ProcessFunction<String, List<String>> { input -> input.toList().map { it.toString() } })
            )
            .apply(Count.perElement<String>())
            .apply(
                    MapElements.into(TypeDescriptors.strings())
                            .via(ProcessFunction { input -> "${input.key} : ${input.value}" })
            )
            .apply(TextIO.write().to("counts"))

    p.run()
}
```

Now you can run the code, and you’ll end up with several files in your project directory of the format **counts-0000\***!

(note: if your main method does not show up as runnable, you may need to right-click pom.xml and “Maven” > “Reimport”)

## What is next?

I plan to use this project as a jumping off point for experimentation, but I also have some small plans for this project on it’s own.

Here is what I’m looking at doing to this repo next:

- Adding tests
- Adding profiles for multiple runners
- Reading from a streaming source of data

Let me know if you’re interested in hearing more about Beam and I will be happy to write more about it.

<iframe allow="accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture" allowfullscreen frameborder="0" name sandbox="allow-scripts allow-same-origin allow-presentation allow-popups allow-popups-to-escape-sandbox" scrolling="no" src="https://www.youtube-nocookie.com/embed/Ycim03b24CU?iv_load_policy=3&amp;modestbranding=1&amp;rel=0&amp;autohide=1&amp;playsinline=1&amp;list=PLWWyzc5ehM91WKA467swlSgR1r_2m5LMJlist%3DPLWWyzc5ehM91WKA467swlSgR1r_2m5LMJ&amp;autoplay=0" width="480" height="270"></iframe>

{% youtube Ycim03b24CU %}
<figcaption>Check out our YouTube channel for more information and live streaming with Beam!</figcaption>