Utilizing Macros & Annotations in a Web Framework (Part 2)
Tags: crystal, metaprogramming
Published: 2020-07-27 17:26:16

---
title: Utilizing Macros & Annotations in a Web Framework (Part 2)
published: true
description: An updated overview of using macros/annotations to build a web framework
tags: crystal, metaprogramming
---

When I first wrote [Utilizing Macros & Annotations in a Web Framework](https://dev.to/blacksmoke16/utilizing-macros-annotations-in-a-web-framework-3abk) I didn't intend on it being a series.  However since its creation, [Athena](https://github.com/athena-framework/athena) has evolved quite a bit and so has my knowledge and understanding of both macros and annotations.  Because of this, I wanted to give an updated look at some of the patterns, approaches, and experiences I used/had with macros and annotations when creating the framework.

Stay tuned for part 3 for a look into how I how annotations and macros to build a compile time service container for Athena's [DependencyInjection](https://athena-framework.github.io/dependency-injection/Athena/DependencyInjection.html) component.

## Registering Routes

In the previous article I mentioned that my ideal syntax would be annotating my controllers with a `@[ART::Controller]` annotation, then iterate over these controller types to build out the routes.  However, after thinking about it more I actually like how it ended up, using an abstract class and iterating over the children of it.  This not only makes it a bit simpler to define a controller as you don't have to remember to add the annotation, but also allows defining common methods within that abstract class that could be useful to all controllers.  When paired with DI, it also makes it possible to define additional abstract controllers that inherit from the default one that could be reused for endpoints requiring similar dependencies/logic.

I also found a workaround to being able to iterate over types with a specific annotation without the need for a parent type or module.  Since all types in Crystal inherit from an `Object` class, we can combine the `all_subclasses` and the `select` methods to get the desired output.

```crystal
annotation MyAnn; end
 
@[MyAnn]
class A; end
 
class B; end
 
@[MyAnn]
class C; end
 
{{Object.all_subclasses.select &.annotation MyAnn}} # => [A, C]
```

## Annotation Data at Runtime

One of the main uses of annotations is to store metadata related to a type, method, or instance variable that is accessible at compile time.  This metadata could then be consumed at compile time to do various things.  However there is not a built in way, such as reflection, to also have access to this data at runtime.  A pattern I started using quite a lot combines the [record](https://crystal-lang.org/api/master/toplevel.html#record(name,*properties)-macro) macro, modules, and the `macro included`hook.

The idea is that you define a module that defines methods within the including type that exposes an array, or some other data structure, of structs representing the data, most likely instance variables, related to that type.  The main benefits of this are:

1. You are using methods, so you are able to access all the instance variables within that type
2. You are using structs, so there is minimal performance overhead
3. You are using modules, so it works for both structs and classes
4. It allows you to use the annotations as a DSL that determine how each struct is instantiated

Lets take a look at an example from Athena's [Serializer](https://github.com/athena-framework/serializer) component.  This component defines an [ASR::PropertyMetadata](https://athena-framework.github.io/serializer/Athena/Serializer/PropertyMetadata.html) struct.  The purpose of this struct is pretty clear; it stores metadata related to a property.  In other words, it describes the state of an instance variable; including its name, type, class it belongs to, etc.  The serializer component also defines the [ASR::Serializable](https://athena-framework.github.io/serializer/Athena/Serializer/Serializable.html) module, similar to `JSON::Serializable`, which defines two main methods: `#serialization_properties` and `.deserialization_properties`. Instead of having the module define methods tied to a specific format, the `ASR::Serializable` module defines generic methods that return arrays of our `ASR::PropertyMetadata` object.  This provides a generic interface that could be consumed in a format agnostic manner.

A example of this concept (cut down for brevity) would look like:

```crystal
# Define an abstract struct to type our methods with
abstract struct PropertyMetadataBase; end

# Define a struct to store information about each instance variable
record PropertyMetadata(IvarType, ClassType) < PropertyMetadataBase,
  name : String,
  type : IvarType.class = IvarType,
  class : ClassType.class = ClassType

# Define an annotation similar to `JSON::Field`'s `skip` option
annotation Skip; end

# Define our module
module Serializable
  macro included
    # :nodoc:
    def self.deserialization_properties : Array(PropertyMetadataBase)
      {% verbatim do %}
        {% begin %}
          {{@type.instance_vars.reject(&.annotation(Skip)).map do |ivar|
              %(PropertyMetadata(#{ivar.type}, #{@type}).new(
                name: #{ivar.name.stringify}
              )).id
            end}} of PropertyMetadataBase
        {% end %}
      {% end %}
    end
  end
end

# Define a type to test with
record User, id : Int32, name : String, age : Int32 do
  include Serializable

  @[Skip]
  @id : Int32
end

pp Example.deserialization_properties # =>
# [
#   PropertyMetadata(String, User)(@name="name", @type=String, @class=User),
#   PropertyMetadata(Int32, User)(@name="age", @type=Int32, @class=User)
# ]
```

The main benefit of this approach is it combines the benefits of using annotations, with having runtime types to expose the same data.  For example, notice how we reject any instance variable that has the `Skip` annotation, and that a `PropertyMetadata` instance isn't created for that property.  The logic that consumes the data from this method need not worry about doing that filtering since it all happens at compile time.

However, there is a limitation with this approach. The logic for handling the `Skip` annotation for example, is hardcoded within the module.  Or in other words, there isn't a way to also expose user defined annotations as part of the metadata struct.  Mainly because there isn't currently way to get all the annotations on a given type, method, or instance variable (see [this issue](https://github.com/crystal-lang/crystal/issues/9322)).  But also there isn't a built in way to represent an annotation (and its data) at runtime.  After a bit of thinking I managed to figure out a pretty decent workaround; use a `record` defined by the user to represent the data read off of their custom annotation.

If you read the [Property Validation](https://dev.to/blacksmoke16/utilizing-macros-annotations-in-a-web-framework-3abk#property-validation) section from my previous article, you would remember how I had to define that hacky `ASSERTIONS` constant so I knew the types of annotations to read, and what properties I should read off of each.  This was because there wasn't a way to get _all_ of the data within an annotation, you had to access each value by name/index using `Annotation#[]`.  Fortunately since that last article,  `Annotation#args` and `Annotation#named_args` methods have been added (by me :wink: [PR](https://github.com/crystal-lang/crystal/pull/7694)).  An example implementation of this would look like:

```crystal
# Define our annotation
annotation MyAnn; end

# Define a record to store the data related to our annotation
record MyAnnConfig, id : Int32, name : String, active : Bool = true

class Foo
  # Apply our annotation to a method,
  # have it return an instance of our struct.
  @[MyAnn(1, name: "Jim")]
  def config : MyAnnConfig
    {% begin %}
      {% ann = @def.annotation MyAnn %}
      MyAnnConfig.new {% unless ann.args.empty? %}{{ann.args.splat}},{% end %}{{ann.named_args.double_splat}}
    {% end %}
  end
end

pp Foo.new.config # => MyAnnConfig(@active=true, @id=1, @name="Jim")
```

There is quite a few noteworthy things to mention here.  The most obvious of which is that we can use default values.  Since the `active` property was not supplied in the annotation, the default value of `true` was used.  We are also able to use both positional and named arguments; positional annotation arguments will be provided as positional arguments and named annotation arguments will be provided as named arguments to the configuration struct.

Another benefit is we get type safety related to the arguments supplied within the annotation.  If we defined our annotation as `@[MyAnn(1, name: "Jim", unknown_value: 192.1)]`, it wouldn't compile with the error:

```crystal
> 1 |
> 2 |
> 3 |       MyAnnConfig.new 1,name: "Jim", unknown_value: 192.1
^--
Error: no overload matches 'MyAnnConfig.new' with types Int32, name: String, unknown_value: Float64

Overloads are:
- MyAnnConfig.new(id : Int32, name : String, active : Bool = true)
```

It also wouldn't compile if it was missing a value that didn't have a default, or if one of the values was not the correct type.  You would also be able to define additional methods on the configuration struct in additional to the standard getters.

We can go one step further to allow for creating our configuration structs more dynamically:

```crystal
# Define our annotation
annotation MyAnn; end

# Define a record to store the data related to our annotation
record MyAnnConfig, id : Int32, name : String, active : Bool = true

# Define a const to hold our annotation types
CUSTOM_ANNS = [] of Nil

# Define a macro to "register" our types
macro register_ann(name)
  {% CUSTOM_ANNS << name %}
end

# Register our annotation
register_ann MyAnn

class Foo
  @[MyAnn(1, name: "Jim")]
  def config
    {% begin %}
      # Iterate over each of our registered annotations
      # assuming the struct name is ann name + "Config"
      {% for ann_name in CUSTOM_ANNS %}
        {% ann = @def.annotation ann_name.resolve %}
        {{ann_name.id}}Config.new {% unless ann.args.empty? %}{{ann.args.splat}},{% end %}{{ann.named_args.double_splat}}
      {% end %}
    {% end %}
  end
end

pp Foo.new.config # => MyAnnConfig(@active=true, @id=1, @name="Jim")
```

This does create a naming convention related to our configuration structs, but it's manageable.  I think ideally the configuration struct would be merged with the `annotation` definition.  This would centralize things to make the annotation itself the type that is applied and that stores the data at runtime.

I had the idea to apply this concept of exposing user defined annotations at runtime to both the serializer and routing components of Athena.  The feature would allow for using more advanced annotation based runtime logic within your [ART::Listeners](https://athena-framework.github.io/athena/Athena/Routing/Listeners.html)/[ART::ParamConverterInterface](https://athena-framework.github.io/athena/Athena/Routing/ParamConverterInterface.html)s and/or [ASR::ExclusionStrategies::ExclusionStrategyInterface](https://athena-framework.github.io/serializer/Athena/Serializer/ExclusionStrategies/ExclusionStrategyInterface.html#annotation-configurations)s.  Since I wanted this feature in two separate shards, I ended up creating a framework of sorts to make defining, registering, and using these annotation configurations easier as well as share code between the shards.

This has since been released as part of the Athena [Config](https://github.com/athena-framework/config) component; which can also be used outside of the Athena ecosystem.  The main concepts are:

*  [ACF.configuration_annotation](https://athena-framework.github.io/config/Athena/Config.html#configuration_annotation(name,*args,&)-macro) - Used to both define and register an annotation, but also register the related configuration struct behind the scenes
* [ACF::AnnotationConfigurations](https://athena-framework.github.io/config/Athena/Config/AnnotationConfigurations.html) - Wraps a hash keyed by annotation class that stores the configuration structs related to each annotation applied to a specific type, method, or instance variable

The wrapping type defines a custom `#[]` method that allows accessing the configuration structs in a type safe way.  Athena's serializer component exposes annotation configurations as part of the `PropertyMetadata` object.  Lets combine some of the examples to demonstrate how it all fits together.

```crystal
# Define an abstract struct to type our methods with
abstract struct PropertyMetadataBase; end

# Define a struct to store information about each instance variable
record PropertyMetadata(IvarType, ClassType) < PropertyMetadataBase,
  name : String,
  annotation_configurations : ACF::AnnotationConfigurations,
  type : IvarType.class = IvarType,
  class : ClassType.class = ClassType

# Define our module
module Serializable
  macro included
    # :nodoc:
    def self.deserialization_properties : Array(PropertyMetadataBase)
      {% verbatim do %}
        {% begin %}
          {{@type.instance_vars.map do |ivar|
              # Use the code from the `ACF::AnnotationConfigurations` type to resolve the applied annotations.
              annotation_configurations = {} of Nil => Nil

              ACF::CUSTOM_ANNOTATIONS.each do |ann_class|
                ann_class = ann_class.resolve
                annotations = [] of Nil

                ivar.annotations(ann_class).each do |ann|
                  pos_args = ann.args.empty? ? "Tuple.new".id : ann.args
                  named_args = ann.named_args.empty? ? "NamedTuple.new".id : ann.named_args

                  annotations << "#{ann_class}Configuration.new(#{ann.args.empty? ? "".id : "#{ann.args.splat},".id}#{ann.named_args.double_splat})".id
                end

                annotation_configurations[ann_class] = "#{annotations} of ACF::AnnotationConfigurations::ConfigurationBase".id unless annotations.empty?
              end

              %(PropertyMetadata(#{ivar.type}, #{@type}).new(
                name: #{ivar.name.stringify},
                annotation_configurations: ACF::AnnotationConfigurations.new(#{annotation_configurations} of ACF::AnnotationConfigurations::Classes => Array(ACF::AnnotationConfigurations::ConfigurationBase)),
              )).id
            end}} of PropertyMetadataBase
        {% end %}
      {% end %}
    end
  end
end

# Define and register our custom annotations
ACF.configuration_annotation MyAnn, id : Int32, active : Bool = true
ACF.configuration_annotation OtherAnn

# Define a type to test with
record User, id : Int32, age : Int32 do
  include Serializable

  @[MyAnn(1)]
  @id : Int32

  @[MyAnn(2, active: false)]
  @age : Int32
end

properties = User.deserialization_properties

properties[0].annotation_configurations[MyAnn].active # => true
properties[1].annotation_configurations[MyAnn].active # => false

properties[0].annotation_configurations[OtherAnn].active # => Error: undefined method 'active' for OtherAnnConfiguration
properties[0].annotation_configurations[OtherAnn]?       # => nil
```

I added an `annotation_configurations` instance variable to our `PropertyMetadata` type, used the code from `ACF::AnnotationConfigurations` create the annotation configurations instance to provide to the metadata object.  I then used the `deserialization_properties` method to access our configurations, using the `annotation_configurations` to get a reference to the wrapping type.  The `#[]` method accepts the annotation class you wish to fetch.  Notice that you get type safety when using getters for the properties you specified when registering the annotation.  `#[]?` is also defined that returns `nil` if the specified annotation was not applied.  A `#has?(type)` method is also available that returns `true` or `false` depending on if any annotations of the provided *type* were applied to that property.  In the end this abstraction layer provides a common standardized interface to use in both shards, as well as makes the required naming convention of the configuration structs an implementation detail.  

This feature greatly increases the flexibility of both the serializer and routing components (or whatever it is implemented within).  An example of how this feature could be used is imagine you define `IgnoreOnCreate` and/or `IgnoreOnUpdate` configuration annotations.  You could then define an exclusion strategy that injects the [ART::RequestStore](https://athena-framework.github.io/athena/Athena/Routing/RequestStore.html) to know what `HTTP` method the current request is.  Logic could then be implemented that would skip properties with the `IgnoreOnCreate` annotation during `POST` requests and skip `IgnoreOnUpdate` properties on `PUT` requests, while still exposing them on `GET` requests.  On the routing side of things, you could imagine a `Pagination` or `RateLimited` annotation used to configure a related event listener.

As usual feel free to join me in the Athena [Gitter](https://gitter.im/athena-frameworkcr/community) channel if you have any suggestions, questions, or ideas. I'm also available on Discord (`Blacksmoke16#0016`) or via [Email](mailto:george@dietrich.app).