Mandelbrot benchmark
Tags: ruby, javascript, kinx, benchmark
Published: 2020-08-19 00:00:16

# Mandelbrot benchmark

- Target ... C, PHP, HHVM, Ruby, Python, PyPy, and our Kinx

## Introduction

I heard PHP8 would support JIT.

Oh, it sounds great! That is also a good situation for our Kinx to show the ability of the performance by `native` keyword in Kinx.

Posted this article means that the result was very good. It is rather over my expectation!

Note that please see the article [here](https://dev.to/krayg/kinx-as-a-script-language-4iko) if you do not know [Kinx](https://github.com/Kray-G/kinx). I will be very happy if you are interested in it.

## Before doing it

#### Benchmark

Look at [here（https://gist.github.com/dstogov/12323ad13d3240aee8f1）](https://gist.github.com/dstogov/12323ad13d3240aee8f1), and you will find some benchmarks. So I made some stuffs based on those.

Note that the environment is quite different. That's why I benchmarked it all. And also I choosen the near version because I can't setup it with exactly same versions, and compared with original one.

But as an original comment said, only PHP did cheat. In fact, it is unnecessary to output the result and see an I/O overhead, so I removed it.

### How to measure the time

It is being used a timer which has been prepared by the language. And, I found the compile time with source code analysis is not included in the measurement. Although it can't be helped about C.

The reason why I have found it is that I felt something strange about HHVM. When it is HHVM, the result time on display is very fast but my actual feeling is different. The result is faster than PHP, but the `real` time of HHVM is slower than PHP as an actual elapsed time.

### Output

I confirmed that all codes were displaying the following output. Our Kinx was also working as we expected.

```
                                       *
                                       *
                                       *
                                       *
                                       *
                                      ***
                                     *****
                                     *****
                                      ***
                                       *
                                   *********
                                 *************
                                ***************
                             *********************
                             *********************
                              *******************
                              *******************
                              *******************
                              *******************
                            ***********************
                              *******************
                              *******************
                             *********************
                              *******************
                              *******************
                               *****************
                                ***************
                                 *************
                                   *********
                                       *
                                ***************
                            ***********************
                         * ************************* *
                         *****************************
                      * ******************************* *
                       *********************************
                      ***********************************
                    ***************************************
               *** ***************************************** ***
               *************************************************
                ***********************************************
                 *********************************************
                 *********************************************
                ***********************************************
                ***********************************************
              ***************************************************
               *************************************************
               *************************************************
              ***************************************************
              ***************************************************
         *    ***************************************************    *
       *****  ***************************************************  *****
       ****** *************************************************** ******
      ******* *************************************************** *******
    ***********************************************************************
    ********* *************************************************** *********
       ****** *************************************************** ******
       *****  ***************************************************  *****
              ***************************************************
              ***************************************************
              ***************************************************
              ***************************************************
               *************************************************
               *************************************************
              ***************************************************
                ***********************************************
                ***********************************************
                  *******************************************
                   *****************************************
                 *********************************************
                **** ****************** ****************** ****
                 ***  ****************   ****************  ***
                  *    **************     **************    *
                         ***********       ***********
                         **  *****           *****  **
                          *   *                 *   *
```

## Benchmark it

It is time to benchmark. First of all, let me introduce source codes.

### C

Here is a version of gcc.

```
$ gcc --version
gcc (Ubuntu 7.4.0-1ubuntu1~18.04.1) 7.4.0
Copyright (C) 2017 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
```

C code is like this.

```c
#include <stdio.h>
#include <sys/time.h>

#define BAILOUT 16
#define MAX_ITERATIONS 1000

int mandelbrot(double x, double y)
{
        double cr = y - 0.5;
        double ci = x;
        double zi = 0.0;
        double zr = 0.0;
        int i = 0;

        while(1) {
                i ++;
                double temp = zr * zi;
                double zr2 = zr * zr;
                double zi2 = zi * zi;
                zr = zr2 - zi2 + cr;
                zi = temp + temp + ci;
                if (zi2 + zr2 > BAILOUT)
                        return i;
                if (i > MAX_ITERATIONS)
                        return 0;
        }

}

int main (int argc, const char * argv[]) {
        struct timeval aTv;
        gettimeofday(&aTv, NULL);
        long init_time = aTv.tv_sec;
        long init_usec = aTv.tv_usec;

        int x,y;
        for (y = -39; y < 39; y++) {
                //printf("\n");
                for (x = -39; x < 39; x++) {
                        volatile int i = mandelbrot(x/40.0, y/40.0);
                        //if (i==0)
                        //      printf("*");
                        //else
                        //      printf(" ");
                }
        }
        //printf ("\n");

        gettimeofday(&aTv,NULL);
        double query_time = (aTv.tv_sec - init_time) + (double)(aTv.tv_usec - init_usec)/1000000.0;
        printf ("C Elapsed %0.3f\n", query_time);
    return 0;
}
```

### PHP/HHVM

It is the version of PHP.

```
$ php --version
PHP 7.2.24-0ubuntu0.18.04.6 (cli) (built: May 26 2020 13:09:11) ( NTS )
Copyright (c) 1997-2018 The PHP Group
Zend Engine v3.2.0, Copyright (c) 1998-2018 Zend Technologies
    with Zend OPcache v7.2.24-0ubuntu0.18.04.6, Copyright (c) 1999-2018, by Zend Technologies
```

It is the version of HHVM.

```
$ hhvm --version
HipHop VM 3.21.0 (rel)
Compiler: 3.21.0+dfsg-2ubuntu2
Repo schema: ebd0a4633a34187463466c1d3bd327c131251849
```

There is no differences of the source code between PHP and HHVM.

```php
<?php
define("BAILOUT",16);
define("MAX_ITERATIONS",1000);

class Mandelbrot
{
    function Mandelbrot()
    {
        $d1 = microtime(1);
        for ($y = -39; $y < 39; $y++) {
            for ($x = -39; $x < 39; $x++) {
                $this->iterate($x/40.0,$y/40.0);
            }
        }
        $d2 = microtime(1);
        $diff = $d2 - $d1;
        printf("PHP Elapsed %0.3f\n", $diff);
    }

    function iterate($x,$y)
    {
        $cr = $y-0.5;
        $ci = $x;
        $zr = 0.0;
        $zi = 0.0;
        $i = 0;
        while (true) {
            $i++;
            $temp = $zr * $zi;
            $zr2 = $zr * $zr;
            $zi2 = $zi * $zi;
            $zr = $zr2 - $zi2 + $cr;
            $zi = $temp + $temp + $ci;
            if ($zi2 + $zr2 > BAILOUT)
                return $i;
            if ($i > MAX_ITERATIONS)
                return 0;
        }

    }


}

$m = new Mandelbrot();
?>
```

### Ruby

It is the version of Ruby.

```
$ ruby --version
ruby 2.5.1p57 (2018-03-29 revision 63029) [x86_64-linux-gnu]
```

Here is the Ruby's source code.

```ruby
BAILOUT = 16
MAX_ITERATIONS = 1000

class Mandelbrot

        def initialize
                #puts "Rendering"
                for y in -39...39 do
                        #puts
                        for x in -39...39 do
                                i = iterate(x/40.0,y/40.0)
                                #if (i == 0)
                                #       print "*"
                                #else
                                #       print " "
                                #end
                        end
                end
        end

        def iterate(x,y)
                cr = y-0.5
                ci = x
                zi = 0.0
                zr = 0.0
                i = 0

                while(1)
                        i += 1
                        temp = zr * zi
                        zr2 = zr * zr
                        zi2 = zi * zi
                        zr = zr2 - zi2 + cr
                        zi = temp + temp + ci
                        return i if (zi2 + zr2 > BAILOUT)
                        return 0 if (i > MAX_ITERATIONS)
                end

        end

end

time = Time.now
Mandelbrot.new
#puts
puts "Ruby Elapsed %f" % (Time.now - time)
```

### Python/PyPy

It is the version of Python.

```
$ python --version
Python 2.7.15+
```

PyPy's version.

```
$ pypy --version
Python 2.7.13 (5.10.0+dfsg-3build2, Feb 06 2018, 18:37:50)
[PyPy 5.10.0 with GCC 7.3.0]
```

Here is the Python's source code. The source code for PyPy is the same.

```python
import sys, time
stdout = sys.stdout

BAILOUT = 16
MAX_ITERATIONS = 1000

class Iterator:
  def __init__(self):
    #print 'Rendering...'
    for y in range(-39, 39):
      #stdout.write('\n')
      for x in range(-39, 39):
        i = self.mandelbrot(x/40.0, y/40.0)

        #if i == 0:
          #stdout.write('*')
        #else:
          #stdout.write(' ')

  def mandelbrot(self, x, y):
    cr = y - 0.5
    ci = x
    zi = 0.0
    zr = 0.0
    i = 0

    while True:
      i += 1
      temp = zr * zi
      zr2 = zr * zr
      zi2 = zi * zi
      zr = zr2 - zi2 + cr
      zi = temp + temp + ci

      if zi2 + zr2 > BAILOUT:
        return i
      if i > MAX_ITERATIONS:
        return 0

t = time.time()
Iterator()
print 'Python Elapsed %.02f' % (time.time() - t)
```

### Kinx/Kinx(native)

Here is the version of Kinx.

```
$ kinx -v
kinx version 0.9.2
```

It is the source code of normal Kinx.

```coffeescript
const BAILOUT = 16;
const MAX_ITERATIONS = 1000;

function mandelbrot(x, y) {
    var cr = y - 0.5;
    var ci = x;
    var zi = 0.0;
    var zr = 0.0;
    var i = 0;

    while (true) {
        i++;
        var temp = zr * zi;
        var zr2 = zr * zr;
        var zi2 = zi * zi;
        zr = zr2 - zi2 + cr;
        zi = temp + temp + ci;
        if (zi2 + zr2 > BAILOUT)
            return i;
        if (i > MAX_ITERATIONS)
            return 0;
    }
}


var tmr = new SystemTimer();
var x,y;
for (y = -39; y < 39; y++) {
    #System.print("\n");
    for (x = -39; x < 39; x++) {
        var i = mandelbrot(x/40.0, y/40.0);
        #if (i==0)
        #    System.print("*");
        #else
        #    System.print(" ");
    }
}
#System.print("\n");
System.print("Kinx Elapsed %0.3f\n" % tmr.elapsed());
```

Here is the source code of Kinx with native. The type is not necessary when it can be assumed from the result of expectation, so that was okay only with adding `:dbl` to the argument.

```coffeescript
const BAILOUT = 16;
const MAX_ITERATIONS = 1000;

native mandelbrot(x:dbl, y:dbl) {
    var cr = y - 0.5;
    var ci = x;
    var zi = 0.0;
    var zr = 0.0;
    var i = 0;

    while (true) {
        i++;
        var temp = zr * zi;
        var zr2 = zr * zr;
        var zi2 = zi * zi;
        zr = zr2 - zi2 + cr;
        zi = temp + temp + ci;
        if (zi2 + zr2 > BAILOUT)
            return i;
        if (i > MAX_ITERATIONS)
            return 0;
    }
}


var tmr = new SystemTimer();
var x,y;
for (y = -39; y < 39; y++) {
    #System.print("\n");
    for (x = -39; x < 39; x++) {
        var i = mandelbrot(x/40.0, y/40.0);
        #if (i==0)
        #    System.print("*");
        #else
        #    System.print(" ");
    }
}
#System.print("\n");
System.print("Kinx(native) Elapsed %0.3f\n" % tmr.elapsed());
```

## Result

Here is the result. It is an average of 10 times. The order is the faster one is upside. 'real' is the result of time command.

|       language     | version  | time(sec) | time(real) |
| ---------------- | :--------: | --------------------: | ------------------: |
| C                |   7.4.0    |                 0.018 |               0.046 |
| PyPy             |   5.10.0   |                 0.020 |               0.122 |
| **Kinx(native)** |   0.9.2    |             **0.048** |           **0.107** |
| HHVM             |   3.21.0   |                 0.068 |               0.552 |
| PHP              |   7.2.24   |                 0.182 |               0.241 |
| Ruby             |   2.5.1    |                 0.365 |               0.492 |
| **Kinx**         |   0.9.2    |             **0.393** |           **0.457** |
| Python           |   2.7.15   |                 0.564 |               0.601 |

Good! Kinx(native) is faster than HHVM. And I am happy because Kinx normal is almost same as Ruby VM which I feel very fast.

By the way, PyPy is too fast, but the time of real is almost same. I guess the difference is optimization.

The result shows HHVM is slower than PHP in the `real` of time command. It was because the compilation time would be long. It can not be helped because it is a spec of language. The same penalty of compilation is shown also in Kinx native, although a little bit.

Okay, let's compare with the result in [the orginal article](https://gist.github.com/dstogov/12323ad13d3240aee8f1). In this benchmark, the differences between environments seem to be a big factor. Only for HHVM, it is strange... but I don't know why. In other cases, it is around 2x faster on my environment.

|       language       | version  | time(sec) | Original result | Original version |
| ---------------- | :--------: | --------------------: | ---------: | :--------------: |
| C                |   7.4.0    |                 0.018 |      0.022 |      4.9.2       |
| PyPy             |   5.10.0   |                 0.020 |            |                  |
| **Kinx(native)** |   0.9.2    |             **0.048** |            |                  |
| HHVM             |   3.21.0   |                 0.068 |      0.030 |      3.5.0       |
| PHP              |   7.2.24   |                 0.182 |      0.281 |        7         |
| Ruby             |   2.5.1    |                 0.365 |      0.684 |      2.1.5       |
| **Kinx**         |   0.9.2    |             **0.393** |            |                  |
| Python           |   2.7.15   |                 0.564 |      1.128 |      2.7.8       |

## Conclusion

Benchmarking is very fun, when it is a good result. I couldn't touch `native` these days, but it is one of characteristic of Kinx, so I want to grow it.

See you next time.

---

By the way, here is the script to measure it. I used [Process](https://dev.to/krayg/kinx-library-process-4h5l) implemented recently. I used a result shown at `average`.

```coffeescript
using Process;

var count = 10;
var command = [$$[1], $$[2]];
var r = [];
var re = /[0-9]+\.[0-9]+/;

for (var i = 0; i < count; ++i) {
    var result = "";
    var [r1, w1] = new Pipe();
    var p1 = new Process(command, { out: w1 }).run();
    w1.close();
    while (p1.isAlive() || r1.peek() > 0) {
        var buf = r1.read();
        if (buf.length() < 0) {
            System.println("Error...");
            return 1;
        } else if (buf.length() > 0) {
            result += buf;
        } else {
            // System.println("no input...");
        }
    }

    re.reset(result);
    if (re.find()) {
        r.push(Double.parseDouble(re.group[0].string));
    }
}
var total = r.reduce(&(r, e) => r + e);
System.println("total  : %8.3f" % total);
System.println("count  : %8d" % r.length());
System.println("average: %8.3f" % (total / r.length()));
```

Thank you!