Groovy Gotchas - Loops, Closures, Scope, and Jenkins DSLs
Tags: groovy, jenkins, closures, jenkinsfile
Published: 2022-05-12 10:54:08

---
title: Groovy Gotchas - Loops, Closures, Scope, and Jenkins DSLs
published: true
description: Deferred execution comes with a bunch of gotchas when you combine loops and closures - but it makes the language that much more interesting...!
tags: groovy, jenkins, closures, jenkinsfile
cover_image: https://live.staticflickr.com/7045/6925760897_4b62206f36_c.jpg
---

<small>([Image (C) Tai Kedzierski](https://www.flickr.com/photos/ducakedhare/6925760897/))</small>

I was trying to understand how some syntax in Groovy was working and it led me to some better understanding on how closures can be used. I'm sure it's all in the documentation but... well I'm the kind of person who really needs to see it in place to understand...

## List and map "comprehensions"

Borrowing from Python, I was trying to understand how to perform "list comprehension" and "dict comprehension", but in Groovy. [An article I found][kerflyn] did give me the answer, and I noticed something a a little odd:

```groovy
// List comprehension to double the values of each item
my_list = my_list.collect({ item ->
 item * 2 // the last statement of the block is the item that gets "returned" for each item
})

// Map comprehension to add ".txt" suffix to each entry
// EDIT - from some _very_ old (pre-2012) example code prior to Groovy 1.7.9
my_map = my_map.inject([:]) { new_map, item ->
    new_map[item.getKey()] = "${item.getValue()}.txt"
    new_map // the returnable must be the final statement
}

// EDIT - there is a better way to do map comprehension in current (post-2022) Groovy:
my_map = my_map.collectEntries { item -> [item.getKey(), "${item.getValue()}.txt"] }
```

(for details, do see [the original post][kerflyn])

Two things that I noticed:

* In the list comprehension, the curly block is a direct argument to `.collect()` - but in the map comprehension, the curly block is placed _after_ the call to `.inject()`
* In both cases, the block looks suspiciously like the `.each { name -> operations; }` notation

So I did what any system-dismantling child would do... and tried something ...

```groovy
def myfunc(message, thing) { println thing }

myfunc("Hello") { println "other action" }
```

This showed me that `thing` did actually populate.... with a _closure_ !

```
basic_closure$_run_closure1@76f2bbc1
```

And thus my journey down to Wonderland began ...

## `.each { }` is not a true loop

Notationally, it looks like using the `each { }` operation on an iterable item results in a loop that steps through each item.

Well, not really: the `each()` function (which, yes, it is) in fact takes one argument: a function handler, and calls it once for each item in the iterable it operates from.

This is why this will not work:

```groovy
def error_on_null(my_sequence) {
    my_sequence.each { item ->
        if(item == null)
            break // Groovy will tell you that "break" must be used in a loop!
    }
}
```

If we re-write this in more familiar constructs, this is what is actually happening:

```groovy
def error_on_null(my_sequence) {
    for(item in my_sequence) {
        null_check(item)
    }
}

def null_check(item) {
    if(item == null)
        { break } // Obviously wrong!
}
```

We can see more explicitly in this way what is happening: the content that checks null-ness is actually in a block and scope of its own - it does not incorporate its own loop, and so it is a syntactic error to try to use `break` there.

When using curly braces for a code block, we are actually defining an anonymous function, and passing it along to the `each()` function which itself implements the loop. This anonymous function is [what is known in Groovy as a _closure_][groovy-docs], a piece of code declared in one scope, and executed anywhere else, probably at a deferred time.

Similarly, with `.collect( {} )` we are passing a closure, that can then be called by `.collect`'s internal logic.

## Closure parameters

Closures can have parameters too.

```groovy
def greet = { greeting, name ->
  println "$greeting , $name"
}

greet("Hello", "Tam")
```

And that's how you get the `name ->` notation in the `.each { }` call we are so much more familiar with.

Passing in closures can be done with multiple syntactical omissions:

```groovy
// Demo function
def call_me(clos) { clos(); }

// All of these are equivalent !

// Explicit param
call_me({ println "Hi";})

// Suffix notation
call_me() { println "Hi";}

// parenthesis omission, since it's the only argument
call_me { println "Hi";}
```

## Function calling

In fact, function calling in Groovy can take many different forms as well.

```groovy
// These are NOT the same

my_func("alpha", "beta") // (A)

my_func(a="alpha", b="beta") // (B)

my_func a: "alpha", b: "beta" // (C)
```

Cases A and B require the function to be defined as:

```groovy
def my_func(a, b) { println "$a -> $b" ; }
```

However case C requires a mapping:

```groovy
def my_func(Map args) {
    a = args.get("a")
    b = args.get("b") // you can also use `.getOrDefault(key, default_value)`
}
```

Also this does not behave as you would expect it:

```groovy
def my_func(a, b) { println "$a -> $b" ; }

my_func(a="alpha", b="beta")

my_func(b="beta", a="alpha")
```

The output of this is...

```
alpha -> beta // OK
beta -> alpha // ??
```

The second instance should not have been reversed right? Well, it is - the arguments continue to be passed in by _position_ and not by name. I am unsure of whether that's a bug or not (certainly feels like one) , but it means that if you need to provide default values for options, you MUST use the `Map` form of the arguments.

## Domain Specific Language: Jenkinsfile

I always did wonder how Jenkinsfile pipelines declared its own code blocks like

```groovy
stage("Build stuff") {
    sh "make clean && make && make install"
}
```

It turns out, the `stage()` function is defined something like this

```groovy
def stage(stage_name, operation) {
    org.hudson.etc.setStageName(stage_name) // for example
    operation()
}
```

When `stage()` is called in my Jenkinsfile, it receives the closure I supply after it as an operation to perform. My closure (the build steps) has access to the variables and namespace in the rest of the file - and so can be passed along to the Jenkins-level `stage()` function which proceeds then to calling it (probably wrapped around some more complex error-handling logic).

## The Closure Gotcha

[Previously I posted][surely-bug] about a behaviour I did not understand where variables were seemingly interpolated at the very last possible moment. I originally thought it had something to do with [interpolation of GStrings][gstring-so] -- but nay!

The reason was, of course, because of closures !

I managed to distil the issue down to the following snippet:

```groovy
operations = []

for(x in [0,1,2]) {
    println "Setting operation $x"
    operations[x] = { println "Running operation $x"; }
}

println "===="

for(op in operations) { op(); }

```

The output:

```
Setting operation 0
Setting operation 1
Setting operation 2
====
Running operation 2
Running operation 2
Running operation 2
```

The reason is that the closure evaluates at a deferred time, with knowledge of `x` _at the time of execution_ - which is after the loop has completed, and so the value at execution time ends up being its last value from the loop in every case ...!

In my solution in my complaint post, I moved the closure out a function, which resulted in it taking the value with which the function was called - which remains constant for each call, and is not affected by the loop.

Finally, a mystery solved!

### Currying the closure

Another way around the problem of deferred resolution would be to curry the closure - call its `.curry()` method to produce a new copy of the closure.

```groovy
for(x in [0,1,2]) {
    operations[x] = { y -> // take an argument
        println "Running operation $y"
    }.curry(x) // immediately feed the outer variable into the argument
}
```

What we are doing here is defining a closure with 1 argument, and calling the curry function with the intended value for it. This returns a new closure, which is the effectively pre-seeded closure

## Extra Gotcha: the top-level run method

Another item I found whilst trawling for answers recently is [how "global" variables work][globals] in a Groovy file script, and how code not encapsulated in a function relates to variables equally not encapsulated.

What looks like code that exists within a self-same scope is actually not quite true ...

In essence:

```groovy
// runtime.groovy
implicit_property = "Available everywhere"
def implicit_local = "Not available inside functions"

println "Top level"
println implicit_property
println implicit_local

def a_func() {
  println "In function"
  println implicit_property // OK
  println implicit_local // Fail - it is "local" to the main "run()"
}

a_func()
```

Groovy compiles to Java and so the above actually ends up looking like this:

```java

// assume "println" is a thing

class runtime { // from the filename - "runtime.groovy"
  static String implicit_property // Declaration separate from assignment

  public void run() { // Called at runtime by Groovy evaluator
    implicit_property = "Available everywhere"
    String implicit_local = "Not available inside functions"

    println "Top level"
    println implicit_property
    println implicit_local

    a_func()
  }

  public void a_func() {
    println "In function"
    println implicit_property // OK
    println implicit_local // Fail - it is "local" to the main "run()"
  }
}
```

This outputs

```
Top level
Available everywhere
Not available inside functions
In function
Available everywhere
Caught: groovy.lang.MissingPropertyException: No such property: implicit_local for class: runtime
groovy.lang.MissingPropertyException: No such property: implicit_local for class: runtime
        at runtime.a_func(runtime.groovy:11)
        at runtime.run(runtime.groovy:14)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
```

.... see what happened...? 😱

The Groovy interpreter first cross-compiles to JVM-compatible code - which puts all the "naked code" (not encapsulated in a function) in an _instance_ "run()" method of an implicit class. It then instantiates the implicit class, and calls this `run()` method.

Note that

* the `implicit_local` only has scope within the run() method and is _not_ available to the other methods
* the `implicit_property` is only populated if the `run()` method executes (it is possible in normal groovy to load a file and not execute `run()` with a `GroovyShell().parse()` action)


### Loading files: `return this`

When using the Jenkinsfile `load` instruction, you need the `return this` final line to be able to call the defined methods

```groovy
// hello.groovy
def say_hi(name) { println name }

return this
```

```groovy
// main.groovy
greeter = load "hello.groovy"

greeter.say_hi("Sam")
```

This is due again to the compile-step pushing the body code into a `run()` function. When `load` executes the file, the body code (`run()`) is executed immediately , and its return value is assigned from the import.

This is, in a nutshell, how Jenkinsfile's `load()` function actually works (as actual Groovy code - try it!):

```groovy
// Behaves pretty much like Jenkins's own `load` operation
def load(path) {
    // This will simply compile the file and create an instance
    //  of the implicit class
    def loaded_file = new GroovyShell().parse(new File(path))

    // And then the <instance>.run() function is actually called separately
    //  hence why we need "return this" at the bottom of the file
    return loaded_file.run()
}
```

## Conclusion

A dive down this rabbit-hole finally allowed me to make sense of a core part of the Groovy language , and identify a very interesting gotcha.

This feels more of a symptom of closures being very implicit in Groovy - and possibly of the false similarities in notation between function declarations, and objects - that closure in my Jenkinsfile looked to me like an "object" in JavaScript (think JSON notation) and as such I had not expected it to be the source of the deferred execution.

Another day, another lesson.

[kerflyn]: https://kerflyn.wordpress.com/2010/06/25/list-comprehension-in-groovy/
[surely-bug]: https://dev.to/taikedz/jenkins-pipelines-surely-this-is-a-bug-2fh8
[gstring-so]: https://stackoverflow.com/questions/70021211/groovy-gotcha-string-interpolation-does-not-always-execute-e-g-in-list-con
[globals]: https://stackoverflow.com/a/6315609/2703818
[groovy-docs]: https://groovy-lang.org/closures.html