How CRuby decides an `if` is a modifier
Tags: ruby, parser
Published: 2021-02-06 16:27:15

---
title: How CRuby decides an `if` is a modifier
published: true
description: Reading Ruby's source code to find the difference between two types of `if`
tags: ruby, parser
//cover_image: https://direct_url_to_image.jpg
---
Ruby has two styles to write `if`.

1. `if foo then bar end`
2. `foo if bar`

This reads natural to human, but not to machines. For example, can you tell if this code is valid or not?

```rb
p if 1 then 2 else 3 end
```

The answer is:

```
$ ruby -e 'p if 1 then 2 else 3 end'
-e:1: syntax error, unexpected `then', expecting end-of-input
```

Because the `if` here is recognized as "modifier if", not "keyword if". So how does Ruby decides the type of `if`?

## parse.y

The answer should be in the [parse.y](https://github.com/ruby/ruby/blob/94fb62e4a01975d61ad074f882068fdd7d6ed474/parse.y), which defines Ruby's grammer.

In the parse.y, you see [keyword_if](https://github.com/ruby/ruby/blob/94fb62e4a01975d61ad074f882068fdd7d6ed474/parse.y#L3241) and [modifier_if](https://github.com/ruby/ruby/blob/94fb62e4a01975d61ad074f882068fdd7d6ed474/parse.y#L1481). It means the type of `if` is decided by the lexer, not the parser.

## lex.c.blt

By grepping `modifier_if`, you will find [lex.c.blt](https://github.com/ruby/ruby/blob/94fb62e4a01975d61ad074f882068fdd7d6ed474/lex.c.blt#L249) has a table of keywords in the function `rb_reserved_word`.

```c
#line 31 "defs/keywords"
      {gperf_offsetof(stringpool, 33), {keyword_if, modifier_if}, EXPR_VALUE},
```

## parse.y

The lexer starts from [yylex](https://github.com/ruby/ruby/blob/94fb62e4a01975d61ad074f882068fdd7d6ed474/parse.y#L9784). It calls `parser_yylex`, which handles the symbols like `+`, `-`, etc. If the character is not a symbol, `parse_ident` is called.

`parse_ident` checks if a keyword begins from the current position with [`rb_reserved_word`](https://github.com/ruby/ruby/blob/94fb62e4a01975d61ad074f882068fdd7d6ed474/parse.y#L9017). The returned `kw` is a member of the table we've seen in `lex.c.blt`.

```c
	/* See if it is a reserved word.  */
	kw = rb_reserved_word(tok(p), toklen(p));
```

In the case of `if` keyword, `kw->id[0]` corresponds to keyword_if and `kw->id[1]` corresponds to modifier_if.

Actually `id` has two values to distinguish keywords and modifiers. According to lex.c.blt, Ruby has five modifiers.

- x if y
- x unless y
- x while y
- x until y
- x rescue y

## When an `if` is a modifier

This is the condition that distinguishes keyword_if and modifier_if. In short, an `if` is a keyword if the lexer state is `EXPR_BEG`; otherwise, it is a modifier.

```c
            if (IS_lex_state_for(state, (EXPR_BEG | EXPR_LABELED)))
                return kw->id[0];
            else {
                if (kw->id[0] != kw->id[1])
                    SET_LEX_STATE(EXPR_BEG | EXPR_LABEL);
                return kw->id[1];
            }
```

## The lexer state

Among the states of the lexer, EXPR_BEG, EXPR_END and EXPR_ARG are the most important. They decides operators like `+`, `-` is unary or binary. For example:

- `1 - 2`: This is binary minus because the state is EXPR_END after the `1`.
- `foo(-1)`: This is unary minus because the state is EXPR_BEG after the `(`.

EXPR_ARG is a bit tricky; On this state, the meaning of `-` changes by the space after it.

- `foo - 1`: binary minus
- `foo -1`: unary minus

What is interesting is that this rule is not so difficult for humans. The former "looks like" binary and the latter "looks like" unary. So you will actually never be bothered by this, unless you are implementing the parser.

## keyword if and modifier if

Now you can tell an `if` is a keyword or modifier by checking the lexer state.

- `foo() if ...`: This is modifier_if because the state is EXPR_END after the `)`.
- `foo(if ...)`: This is keyword_if because the state is EXPR_BEG after the `(`.
- `foo if ...`: This is modifier_if because the state is EXPR_ARG after the ` ` before `if`.

## Why this matters to me

I think most Rubyists does not care about corner cases like this; However I needed to figure out this because I'm making my original programming language [Shiika](https://github.com/yhara/shiika) which has Ruby-like syntax.

As you've seen, parsing Ruby-like syntax is not easy, especially parsing method calls without parentheses. I'm happy if this entry helps someone who want to make a Rubyish language.