Write your own Domain Specific Language in Ruby
Tags: ruby, dsl, metaprogramming, tutorial
Published: 2023-02-06 12:36:00

Let's do some metaprogramming in Ruby! This article assumes you have some familiarity with writing Ruby and are interested in learning how to use metaprogramming.

_This article is also available as a [video recording of a talk](https://tomdebruijn.com/talks/ruby-write-your-own-domain-specific-language/)._

## What's a DSL?

Let's start with some definitions on what we're going to be looking at. The abbreviation _DSL_ stands for _Domain Specific Language_.

A DSL is a sub-language within a Ruby app (or any other programming language) that is specific to a certain domain: like a Ruby gem or part of an app. Without that gem or app, the DSL won't work. It's specific to that domain.

In Ruby there's a gem named [Rake](https://rubygems.org/gems/rake). This gem provides a DSL to create tasks to be run from the command line. A small example looks like this:

```ruby
# Rakefile
task :hello do
  puts "Hello there"
end
```

You can call this task from the terminal with the following command. When called, it will perform the block we've given to the `task` method.

```sh
$ rake hello
Hello there
```

This is the power of a DSL. We don't need to know how to listen to arguments given to Ruby from the command line. Rake does this for us. We only need to define the commands.

## Why use a DSL?

### Configuration

One of the most common reasons to create your own DSL is for configuration, for an app or part of a gem. In Rails, the `Application` class contains configuration for the different gems that make up Rails. It's even possible to define your own [custom config options](https://guides.rubyonrails.org/configuring.html#custom-configuration) specific to your app.

```ruby
# config/application.rb
module MyApp
  class Application < Rails::Application
    config.load_defaults 7.0

    config.time_zone = "UTC"

    config.app.custom_option = "some value"
  end
end
```

Within the Application class, the `config` object can be used to set config options or load defaults, like shown above.

### Automate writing code

A DSL can be used to abstract away a bunch of repetitive code. In a Rails app there can be many different HTTP endpoints the app listens to. To write a Ruby app that listens to these endpoints can require a lot of code. In Rails, we can declare several different routes with one method: `resources`

```ruby
Rails.application.routes.draw do
  resources :posts do
    resources :comments
  end
  root "homepage#show"
end
```

The `resources` method will do several things:

- create several endpoints like `GET /posts`, `GET /posts/new`, `POST /posts`, `DELETE /posts`, etc.;
- nest endpoints, if they are nested within another `resources` method call's block, like in the example above for `comments`;
- route the request data on the endpoint to the relevant controller.

This is all defined with one method call. We don't need to go through several files and methods to make sure these steps are all done for every route. This makes, what would be high churn code, more maintainable*.

_*: At times, we may be swapping out a lot of code with less code, but that code is more complex because of the metaprogramming involved._

## Writing your own Ruby DSL

### The start of a DSL

The first thing that usually gets a DSL is configuration. A gem ships with a configuration class or a part of an app does. A new `Config` class gets created with some config options. (In this case a CLI tool that has options to print more information with `verbose` and a particular output format called `documentation`.)

```ruby
class Config
  def initialize(options = {})
    @options = options
  end
end

config = Config.new(
  :verbose => true,
  :format => :documentation
)
# => #<Config:...
#        @options={:verbose=>true, :format=>:documentation}>
```

The above example is usually enough for a small component.

Let's look how we can make this DSL feel more like some of the DSLs we've seen. In the example below we don't configure the Config class with a Hash of options, but use method calls like the `verbose=` attribute writer.

Ruby has helpers for defining reader and writer methods on a class for instance variables. Calling `attr_accessor :verbose` in a class definition will define the `verbose` or `verbose=` methods.

```ruby
class Config
  attr_accessor :verbose, :format
end

config = Config.new
config.verbose = true
config.verbose # => true
config.format = :documentation

pp config
# => #<Config:...
         @format=:documentation,
         @verbose=true>
```

While this is quick to set up, it has some downsides. Like before, we want to store all config options on a `@options` Hash instead of on their own instance variables. To export the config options as a Hash, we would need to export them all manually. Now there are more places that need to be updated the more config options are added. Easily forgotten, easily broken.

```ruby
class Config
  # ...

  def options
    {
      :verbose => @verbose,
      :format => @format
    }
  end
end

pp config.options
# => {:verbose=>true, :format=>:documentation}
```

To make the Config class work with a Hash of options, we can define our own methods for each config option. In these methods we set the values on the `@options` Hash, using the key of the config option we want to set. Great! We now can export our configuration with ease.

```ruby
class Config
  attr_reader :options

  def initialize
    @options = {}
  end

  def verbose=(value)
    @options[:verbose] = value
  end

  def format=(value)
    @options[:format] = value
  end
end

config = Config.new
config.verbose = true
config.format = :documentation

pp config.options
# => {:verbose=>true, :format=>:documentation}
```

With this setup, we run into a familiar problem. Every time a new config option is added we need to add another method for that config option. As more config options are added, the Config class grows and grows, becoming more difficult to maintain.

## Dynamically define methods in Ruby

Instead of defining a method per config option, we can dynamically define these methods. Dynamically defining methods allow us to quickly define several methods that share almost the same implementation.

In the example below I've created a new class method, as indicated with the `def self.def_options` syntax (1). This method, "define options", receives an array of option names to define methods for (2). In the `def_options` method it will loop through these option names (3). In each iteration of the loop it will call the [`define_method` method](https://ruby-doc.org/core-3.1.2/Module.html#define_method-method) (4). With this method we can define a new method, like with the `def <method name>` syntax, but with a dynamic method name.

```ruby
class Config
  def self.def_options(*option_names) # 1
    option_names.each do |option_name| # 3
      define_method "#{option_name}=" do |value| # 4
        @options[option_name] = value # 5
      end
    end
  end

  def_options :verbose, :format # 2

  def initialize
    @options = {}
  end

  def options
    @options
  end
end

config = Config.new
config.verbose = true
config.format = :documentation

pp config.options
# => {:verbose=>true, :format=>:documentation}
```

The `define_method` method receives the name of the method, and a block (4). This block will be the implementation of the method we are defining. In this case, it will call the `@options` instance variable to receive the Hash of options configured, and add a new value for the option name we've declared (5). The DSL syntax for the end-user remains the same.

## Using Ruby blocks

In my opinion, no Ruby DSL is complete without blocks*. They look good for one, but they also provide a new scope of context to users of our DSL, in which our new DSL rules apply.

_*: It's perfectly fine to have a DSL without blocks._

```ruby
Config.configure do |config|
  config.verbose = true
  config.format = :documentation
end
```

In the next example, I've created a new class method called `configure`. To interact with blocks given to a method, we can use the `yield` keyword. If a method has been given a block, `yield` will call that block. If we give a value to `yield`, it will make it so that the block receives that value as an argument in the block, as is shown with `|config|` in the example above.

```ruby
class Config
  def self.configure
    instance = new # Create a new instance of the Config class
    yield instance # Call the block and give the Config instance as an argument
    instance
  end

  # ...
end
```

### Using instance_eval

We can make this shorter, and in my opinion, feel more like a DSL with its own block context. We can drop the `|config|` block argument and avoid having to call `config.` in front of every config option.

```ruby
Config.configure do
  verbose true
  format :documentation
end
```

_I've omitted the equals sign (`=`) from the example here. More on that later._

To make this work we go back to our `configure` class method. Instead of `yield`, we'll use [`instance_eval`](https://ruby-doc.org/core-3.1.2/BasicObject.html#instance_eval-method). (Eval may sound scary, and it has some things to look out for, but in this small example it's quite safe.)

To call `instance_eval` we need to pass in the block given to the method. We can't use `yield` for this. We need to explicitly specify the block argument of the `configure` method with `&block`. The key part being the ampersand (`&`) of the argument name `&block`, telling Ruby it's the block argument.

```ruby
class Config
  def self.configure(&block)
    instance = new
    instance.instance_eval(&block)
    instance
  end

  # ...
end
```

(In Ruby 3.1 and newer the `&block` argument can also be written as only the ampersand `&`, the name is optional.)

When called on another object, like our Config class instance, `instance_eval` will execute that block within the context of the Config instance. The context of that block is changed from where it was defined to the Config instance. When a method is called within that block, it will call it directly on the Config instance.

> __Local variable assignment gotcha__
>
> Remember how in the example above I omitted the equals sign from the method names? When calling a method name ending with an equals sign in `instance_eval`, Ruby will interpret it as a local variable assignment instead. It will not call the method on the Config instance. That's how Ruby works, and we can't change that. I omitted the equals sign to make the DSL still work.
>
> ```ruby
> Config.configure do
>   # This won't work
>   verbose = true
>   format = :documentation
> end
> ```
>
> It's possible to work around this by explicitly calling the methods on `self.`, but this syntax becomes basically the same as using `yield`. Instead, I removed the equals sign from the method name.
>
> ```ruby
> Config.configure do
>   # This works, but requires `self.` in front of every method call
>   self.verbose = true
>   self.format = :documentation
> end
> ```

### When to use yield or instance_eval?

I've shown two ways of calling blocks in Ruby. Which one should be used is up to the authors of the DSL. I have my personal preference for using `instance_eval`, but there are definitely scenarios where using `yield` works better.

When something needs to be configured, using a lot of methods with an equals sign (writer methods), using `yield` with a block argument is quite common.

```ruby
Config.configure do |config|
  config.verbose = true # Set a value
end
```

When defining something or calling actions, I see the `instance_eval` approach used a lot.

```ruby
Config.configure do
  load_defaults! # Perform an action
end
```

That doesn't mean one excludes the other. Gems like [Puma](https://github.com/puma/puma) use the `instance_eval` method for their configuration file.

```ruby
# config/puma.rb
workers 3
preload_app!
```

Use the approach you feel works best for your DSL.

## Share DSL behavior with modules

With the above methods you can start building your DSL. This gives us a good toolbox. As the DSL grows, or more DSLs get added to the domain, it's good to share behavior between the DSL classes.

For example, this domain has two separate configuration classes: a Config and an Output class. One configures how the gem works, the other how it outputs results to the terminal.

```ruby
# Multiple configuration classes
Config.configure do
  enabled true
end

Output.configure do
  verbose true
  format :documentation
end
```

We don't want to repeat ourselves, so we move the DSL definition behavior to a module called Configurable. When that module is included the class method `def_options` is made available with the class definition.

```ruby
class Config
  include Configurable

  def_options :enabled
end

class Output
  include Configurable

  def_options :verbose, :format
end
```

When the Configurable module is included, it adds the `options` method to the class. The module also extends the class it's included in, using the [`included` callback in Ruby modules](https://ruby-doc.org/core-3.1.2/Module.html#included-method). In that callback, the class is extended with the ClassMethods module, defined inside the Configurable module. This module will add the class methods (`def_options` and `configure`) to the class. This is also how things like [ActiveSupport::Concern](http://api.rubyonrails.org/classes/ActiveSupport/Concern.html) work.

```ruby
module Configurable
  def self.included(base)
    base.extend ClassMethods
  end

  module ClassMethods
    def def_options(*option_names)
      # ...
    end

    def configure(&block)
      # ...
    end
  end

  def options
    @options ||= {}
  end
end
```

With all the logic moved to the Configurable module, we now have a reusable module to create as many config classes as the domain needs.

## Nesting DSLs

The Output class is really a sub domain of the Config class though. It's part of the gem config: the configurations should be nested within one another.

```ruby
Config.configure do
  enabled true

  output do
    verbose true
    format :documentation
  end
end
```

Rather than Output being a top-level config class, we store it on the Config class. In the `output` method in the example below, the method creates a new Output instance. When a block is given, it will `instance_eval` the block on the `@output` object. The method will always return the created `@output` object so the sub-config can be accessed.

```ruby
class Config
  include Configurable

  def_options :enabled

  class Output
    include Configurable

    def_options :verbose, :format
  end

  def output(&block)
    @output ||= Output.new
    @output.instance_eval(&block) if block_given?
    @output
  end
end
```

## Module builder pattern

The Module builder pattern is a really neat design pattern in Ruby that allows us to do away with the two step process of including a module and then calling methods included by it. This pattern is described in more detail in [The Ruby Module Builder Pattern by Chris Salzberg][Module builder pattern].

In the example below, a new `ConfigOption` module is used to include a dynamically defined module. For the end-user, the resulting DSL remains the same.

```ruby
class Config
  include ConfigOption.new(:verbose, :format)
end
```

When `ConfigOption.new` is called, the desired config option names are given to the `initialize` method. Like before, we iterate over this list. Using `define_method` the necessary methods are defined on the module. It's possible to do so in this `initialize` method, because it's creating a new implementation of the ConfigOption module.

```ruby
class ConfigOption < Module
  def initialize(*option_names)
    define_method :options do
      @options ||= {}
    end

    option_names.each do |option_name|
      define_method option_name do |value|
        options[option_name] = value
      end
    end
  end
end
```

For a much more in-depth look, I can highly recommend reading [The Ruby Module Builder Pattern by Chris Salzberg][Module builder pattern].

## Create DSL objects

Another step I recommend is to separate the behavior of the DSL itself and the actual app code that uses it, by creating DSL classes. These classes are only used when the end-user interacts with the DSL, like configuring a gem. When the configuration is done, the options are read from the DSL class and set on whatever config object the gem uses.

Like before, we have a Config class. When configured using `Config.configure`, it creates a ConfigDSL class (1). This ConfigDSL class will become the context of the block given to the `configure` class method.

```ruby
class Config
  def self.configure(&block)
   dsl = ConfigDSL.dsl(&block) # 1
   new(dsl.options) # 2
  end

  attr_reader :options

  def initialize(options)
    @options = options
  end
end

class ConfigDSL
  include ConfigOption.new(:verbose, :format)

  def self.dsl(&block)
    instance = new
    instance.instance_eval(&block)
    instance
  end
end
```

After the `configure` block has been called, the Config class reads the options from the ConfigDSL and initializes a new instance of itself with these options (2).

With this approach the DSL that configures the gem is only used when the app starts. The app doesn't carry around all that extra weight of how the DSL works all the time. The separation of this behavior makes it easier to not accidentally call the configure DSL when it should not be available.

This approach also solves a downside of using `instance_eval`. When calling blocks this way, private methods are accessible inside the block. That's something we usually don't want to allow.

```ruby
class Config
  def self.configure(&block)
    instance = new
    instance.instance_eval(&block)
    instance
  end

  private

  def secret_method
    "super secret method"
  end
end

Config.configure do
  # This should raise an error about calling
  # a private method, but it doesn't
  secret_method
  # => "super secret method"
end
```

Private methods can be accessed in blocks called using `instance_eval`, because the block is evaluated in the context of the instance. It's as if it's being run from a method within that object. With separate DSL classes you have to worry less about private methods that you don't want anyone to call.

## Conclusion

Now that our toolbox is complete we can create our own Domain Specific Language using Ruby. We have the following tools at our disposal:

- Use `define_method` to dynamically define new methods on classes.
- Use Ruby blocks to give your DSL that real DSL feeling and create a context wherein your DSL shines.
    - Use `yield` to return a DSL object as a block argument, or;
    - Use `instance_eval` to change how to block works, and allow end-users to directly call methods on the new context of new block.
- Use Modules to:
    - share behavior between many classes, and;
    - use the [Module builder pattern] to remove any logic of how the DSL is constructed from the class that uses the DSL.
- DSL objects separate the logic of how the DSL works and how the rest of the gem works. Creating a better separation of responsibilities in your code.

Now go forth, and build your own DSL!

[Module builder pattern]: https://dejimata.com/2017/5/20/the-ruby-module-builder-pattern