Write test cases for your Apache Logstash filters
Tags: logstash, rspec, docker, ruby
Published: 2018-08-14 11:47:14

---
title: Write test cases for your Apache Logstash filters
published: true
description: RSpec test cases for your Apache Logstash filters
tags: logstash, rspec, docker, ruby
---

About 1 year ago I started to work on ELK(ElasticSearch, Logstash, Kibana) setup for BI platform. While I was working on processing the events coming to Logstash I found that it's pretty annoying to work on filter changes if you have complex regular expressions, structures or conditions.

I had a lot of experience in Ruby On Rails development the last years and decided to cover Logstash filters by RSpec test cases to save my time, because logstash is using jruby inside of their packages and gems to distribute the extensions.


Also it would be great to have everything in docker to skip Logstash and gem installations on local machine.


Lets start from the Dockerfile and then I will extend step by step the project structure.

```Dockerfile
FROM logstash:2.4

# Install Rspec related dependencies
RUN logstash-plugin install --development

# Install prod dependencies
RUN logstash-plugin install logstash-filter-prune

ARG ES_PLUGIN=logstash-output-elasticsearch-6.2.4-java.gem
ARG KAFKA_PLUGIN=logstash-input-kafka-7.0.0.gem

COPY gems/${ES_PLUGIN} /tmp/${ES_PLUGIN}
RUN logstash-plugin install /tmp/${ES_PLUGIN}

COPY gems/${KAFKA_PLUGIN} /tmp/${KAFKA_PLUGIN}
RUN logstash-plugin install /tmp/${KAFKA_PLUGIN}
```

`gems/` contains freezed gem versions for logstash setup.


Faster way to run test cases lets define Makefile:

```Makefile
NAME = your_logstash

build:
	docker build -t $(NAME) .
.PHONY: build

clean:
	docker rmi --force $(NAME)
.PHONY: clean

test:
	@docker run --rm -t -i \
		-v `pwd`/../:/app \
		-w /app \
		$(NAME) \
		/bin/bash -c "rspec /app/logstash/spec/$(TEST_CASE)"
.PHONY: test

console:
	@docker run --rm -t -i \
		-v `pwd`/../:/app \
		-w /app \
		$(NAME) \
		/bin/bash
.PHONY: console
```

`make test` - run rspec inside of your container
`make console` - opens interactive terminal inside of docker container in case if you want to run everything manually and debug(`binding.pry`) your specs.

Now it would be great to start working on `spec/spec_helper.rb` that's root of rspec.

```ruby
require "logstash/devutils/rspec/spec_helper"
require 'rspec'
require 'rspec/expectations'

require 'ostruct'
require 'erb'
require 'yaml'
require 'json'

# running the grok code outside a logstash package means
# LOGSTASH_HOME will not be defined, so let's set it here
# before requiring the grok filter
# coming from the original examples for logtash specs.
unless LogStash::Environment.const_defined?(:LOGSTASH_HOME)
  LogStash::Environment::LOGSTASH_HOME = File.expand_path("../", __FILE__)
end

module Helpers
  ROOT_PATH = File.dirname(File.expand_path(__FILE__))
  TEMPLATES_PATH = File.join(ROOT_PATH, '..', 'conf.d/')

  def load_fixture(filename, settings = {})
    message = File.read(File.join(ROOT_PATH, 'fixtures', filename))
    settings.merge('message' => message)
  end

  def load_filter(filename, render_vars = {})
    content = File.read(File.join(TEMPLATES_PATH, filename))

    render_vars = OpenStruct.new(render_vars)

    # it's not elegant enough :) but found it as the easiest way to handle
    # jinja2 simple variables replacement.
    template = ERB.new(content.gsub('{{', '<%=').gsub('}}', '%>'))
    template.result(render_vars.instance_eval { binding })
  end
end

require "logstash/filters/grok"
```

In my logstash filters I am using the statements from jinja2 python library `{{ }}` to replace variables on deploy by using ansible.

Now we are ready to define actual spec to test our logstash filter.

lets assume you want to regex line like `username=<username>`. Define your `filter_spec.rb` inside `spec/filters` folder. Filters should not deal with input and output logstash statements. Instead we should read fixture with example of expected line to process and then apply filter.


```ruby
require_relative '../spec_helper'

describe 'elb' do
  extend Helpers

  # set config using our defined filter in `conf.d/01-filter.conf` file.
  config load_filter('01-filter.conf')

  # You can define in spec/fixtures/sample1.txt
  # Example: username=oivoodoo
  sample(load_fixture('sample1.txt', 'type' => 'your-source-data')) do
    insist { subject.get('username') } == 'oivoodoo'
  end
end
```

```conf
filter {
  if [type] == "{{ type }}" {
    grok {
      match => {
        "message" => [
          "%{WORD:username}"
        ]
      }
    }

    prune {
      blacklist_names => [
        "@version",
        "message"
      ]
    }
  }
}
```


I believe it should be enough to test your filters. Found it pretty useful to save time instead of deploying and waiting new coming data. Also it's easy to add `ruby` filter inside of filter and place `binding.pry` and verify `event` object.