Converting a nilable generic argument to its non-nil version
Tags: crystal
Published: 2023-01-07 18:38:42

I have struggled a great deal in the past with making generic code that provides functionality to transform a generic argument from a nilable version to a non-nilable version. And I just learned something pretty cool today.

Currently, I am working on a parsing and validation library and want to provide the ability to type-safely move from a nilable string input, to a non-nilable output without directly tying the library to only working with strings are optional inputs.

Here's some simple example code

```crystal
class Parser(T)
  private getter input : T

  def initialize(@input)
  end

  def not_nil
    if temp = input
      return temp
    else
      raise "input was supposed to be present, but was nil"
    end
  end
end

input = "raw_input".as(String?)
Parser.new(input).not_nil #=> "raw_input" : String

input = nil.as(String?)
Parser.new(input).not_nil #=> runtime exception!
```

I'm hoping that code example is pretty straightforward. It shows a very basic `Parser` class that has one method called `Parser#not_nil`. It returns the non-nilable version of whatever you give it, otherwise it raises an exception if the input actually is nil. In the example, I gave it a nilable string and it returns a non-nilable string. If you are confused as to why I use a `temp` variable, read more on that [here](https://crystal-lang.org/reference/1.6/syntax_and_semantics/if_var.html#limitations).

One thing to notice about this code example, is that there is no way to express the return type of `Parser#not_nil` so I am forced to let Crystal infer it. It works as expected but I am not overly fond of leaving off return types. It could easily infer something I didn't expect because there is a bug in the code and it also leaves the documentation lacking.

This code is not very good, though. As I was saying, I'm working on a parsing _and validation_ library so instead of an exception, I'd prefer a way to express this so that it results in a value that I can return to the user.

If your familiar with functional languages (or Rust), I'm talking about implementing a Result type of sorts. A Result type is a data structure that can only be represented as one of two options: an `Ok` or an `Err` (or Error to be more explicit). This is implemented generically to where the Result wraps the value contained in either the Ok or Err.
For my purposes, I want the Ok to wrap any value `T` and the Err to always wrap a String. With Crystal generics you'd write the Result as `Result(String, T)` where the first item in the parens is the Err type and the second item is the Ok type. To not continue the article with two generics, and because it more closely matches my actual implementation, I will use a `Validated(T)` type that is essentially equal to a  `Result(String, T)` and has two implementing types of `Valid` and `Invalid`.

Here's a real basic implementation of the Validated class.

```crystal
abstract class Validated(T)
  class Valid(T) < Validated(T)
    getter value : T
    def initialize(@value)
    end
  end

  class Invalid(T) < Validated(T)
    getter value : String
    def initialize(@value)
    end
  end
end
```

There's very little code here, but it's the basic implementation that matter. I know it looks weird for `Validated::Invalid` to also have the generic type on it since it only deals with string values (the error messages), but it relates to the code that is missing and it would be more difficult to illustrate what I'm talking about without it. In particular, the code that is missing, involves converting Valid to Invalid and vice-versa. You couldn't do that without keeping up with the valid type on the invalid classes.

_FINALLY_, let's get to the problem code so that I can show off what I learned today.

```crystal
class Parser(T)
  private getter input : T

  def initialize(@input)
  end

  def not_nil
    if temp = input
      Validated::Valid.new(temp)
    else
      Validated::Invalid.new("Expected input to be present, but was nil.")
    end
  end
end

input = "raw_input".as(String?)
Parser.new(input).not_nil.value #=> compilation error!
```

If you actually try to compile this, it fails. It fails because the compiler has no idea what the generic type is supposed to be for the Invalid class. The argument that we pass to create it, is the validation error message after all and we want it to be the not-nil type. We have finally reached the problem that I have run into multiple times and next step has time and again been to open up Discord, go to the Crystal channel, complain, then go do something else while wondering in the back of my mind whether or not I should give up on Crystal and use another language because I figured it was unsolvable. But I decided to do something differently today. I decided to look into where I knew Crystal did something similarly. Specifically, I know that in Crystal you can call `[1, 2, nil, 3].compact` and get back the array `[1, 2, 3]` with a type of `Array(Int32)`. The compiler assures that there can't be nil with the type signature. If the core language is able to express a change from `T?` to `T`, I should be able to as well. So I opened up the core codebase and found that [`Array#compact`](https://github.com/crystal-lang/crystal/blob/5402f6a38f69e45d4c20fa0ee685ffb6f1aa416b/src/array.cr#L727-L729) delegates to [`Enumerable#compact_map`](https://github.com/crystal-lang/crystal/blob/5402f6a38f69e45d4c20fa0ee685ffb6f1aa416b/src/enumerable.cr#L230-L239). Here's the code for that method

```crystal
def compact_map(& : T -> _)
  ary = [] of typeof((yield Enumerable.element_type(self)).not_nil!)
  each do |e|
    v = yield e
    unless v.nil?
      ary << v
    end
  end
  ary
end
```

This took me a while to understand. It's calling this `Enumerable.element_type` function and yielding with it and I still don't understand that part, but the important piece is that it's calling `typeof((...).not_nil!)`.

🤯🤯🤯🤯

I had no idea that you could call `not_nil!` inside `typeof` without raising an error if the thing you are calling it on is actually nil.

```crystal
x = nil.as(String?)
typeof(x.not_nil!) #=> String : Class
```

🤯🤯🤯🤯

So `Enumerable#compact_map` is able to let the compiler infer the return type by creating the return array with the non-nilable type.

Now for the grand reveal of how I was able to use this in my own code.

```crystal
class Parser(T)
  private getter input : T

  def initialize(@input)
  end

  def not_nil
    if temp = input
      Validated::Valid.new(temp)
    else
      Validated::Invalid(typeof(input.not_nil!)).new("Expected input to be present, but was nil.")
    end
  end
end

input = "raw_input".as(String?)
Parser.new(input).not_nil.value #=> "raw_input" : String

input = nil.as(String?)
Parser.new(input).not_nil.value #=> "Expected input to be present, but was nil." : String
```

Hey!! Look at that!! It works!!

So while I use the `temp` to allow the if-block to narrow the type down, I need to avoid doing that with the `typeof((...).not_nil!)` trick so that it remains the original type. The compiler now understands that, while it doesn't have a reference to the not-nil value, Invalid has the same generic type as Valid.

I love this, and I'm going to run with it in my code that does a whole lot more that's outside the scope of this post but I will hopefully have it ready soon to share. I hope this will be helpful to someone else and that your mind is as blown as mine is.

By the way, I have noticed the core team refers to this hack in generic terms as `T -> T!` and would love for this to be implemented in Crystal without the hacks.