Sparky - powerful pocket size task runner server in crontab style
Tags: docker, perl6, linux, devops
Published: 2018-10-19 19:48:38

---
title: Sparky - powerful pocket size task runner server in crontab style
published: true
description: Small but powerful asynchronous cron jobs runner built on Sparrow plugins system.

tags: docker, perl6, linux, devops
---

![Nice web UI](https://raw.githubusercontent.com/melezhik/sparky/master/images/sparky-web-ui3.png) 


Sometimes I need to run some periodical _tasks_ and get _reports_. Strait on my machine. Something simple and configurable. And yeah, sometimes I want to run task **remotely**, not only on localhost, over **ssh** or on **docker** instances. 


Linux provides old good **cron** for that. However if tasks are not just small scripts crontab quickly becomes awkward solution and is hard to maintain. There are "heavy" tools like Jenkins or Team City, but it seems overkill for "one-machine" purpose. I need something light, quick to deploy and hack-able.  


Let me introduce small, but flexible cron jobs runner called - [Sparky](https://github.com/melezhik/sparky).


The essential features of Sparky:

* Defining jobs times in crontab style
* Scenarios defined as [Sparrowdo](https://github.com/melezhik/sparrowdo) scripts
* [Nice set](https://github.com/melezhik/sparrowdo/blob/master/core-dsl.md) of predefined tasks is available
* Everything is kept in SCM repository - easy to port, maintain and track changes
* Tasks gets run **asynchronously** in one of 3 flavours - 1) on localhost 2) on remote machines via **ssh** 3) on **docker** instances
* Nice web UI to read tasks reports is provided 


Interested? Let's go ahead! (:



# Install


Sparky is written on Perl6, so install as Perl6 module, we want the lates GH version:
 
```
$ git clone Sparky https://github.com/melezhik/sparky.git
$ cd sparky
$ zef install .

```

# Initialization

Sparky stores it's data in database, the default one is `sqlite` ( mysql/postgrsql are also possible ), we need create a schema:


```
$ perl6 db-init.pl6

```

Now let's run sparky services, the first one is called `sparky-web` is for UI application and second one called `sparkyd` is daemon to run tasks

The easiest way to do so is using Sparrowdo systemd installers come with distribution:

```
$ sparrowdo --sparrowfile=utils/install-sparkyd-systemd.pl6 --local_mode
$ sparrowdo --sparrowfile=utils/install-sparky-web-systemd.pl6 --local_mode
```

If everything is ok  we will see Sparky UI available at http://localhost:3000 and sparkyd daemon seen as a process in the process list:


```
$ ps uax | grep sparkyd
root       9511  3.3 10.0 574440 202988 ?       Ssl  14:38   7:35 /opt/rakudo-pkg/bin/moar --execname=/opt/rakudo-pkg/bin//perl6 --libpath=/opt/rakudo-pkg/share/nqp/lib --libpath=/opt/rakudo-pkg/share/nqp/lib --libpath=/opt/rakudo-pkg/share/perl6/lib --libpath=/opt/rakudo-pkg/share/perl6/runtime /opt/rakudo-pkg/share/perl6/runtime/perl6.moarvm /opt/rakudo-pkg/share/perl6/site/bin/sparkyd --root=/root/projects/devops/sparky
```

Now the most exciting part - setting up the first task, run it and see the results!


# Sparky task

As has been sain, Sparky task is just a Sparrowdo scenario, if you're not familiar with [Saprrowdo](https://github.com/melezhik/sparrowdo) - don't worry there are a plenty of posts about this tool. But for us is just a handy [DSL](https://github.com/melezhik/sparrowdo/blob/master/core-dsl.md) to create users scenarios:


```

$ nano project1/sparrowfile

#!perl6

bash "echo Hello! My name is Sparky!";

```

This is simple task is ready to get run with Sparky, but first we need to set up crontab entry, it's easy, it's just a good old crontab style:


```
$ nano project1/sparky.yaml

# what about running every minute? 
crontab: "* * * * *"
```


I have not said that, but we should keep 2 files (`sparrowfile` and `sparky.yaml`) in _some_ directory located in `$SPARKY_ROOT` directory which is `~/.sparky/projects` by default:

```

$ ls -d ~/.sparky/projects
project1
project2
project3

```
So sparkyd reads all the sub-directories ( 1 level depth ) located at `$SPARKE_ROOT` and executes found sparky tasks. The `$SPARKE_ROOT` is configurable through environmental variable or as a command line option for sparky task runner daemon:


```
$ sparkyd --root=/path/to/sparky/root/directory
```


Ok, let's go to WEB UI to see the results for our first task:


```
running sparrow tasks on 127.0.0.1 ... 

target OS is - centos7
push [task] run bash: echo 'Hello! My name is Sparky!' ... OK
SPL file /opt/sparky-sparrowdo/project1/sparrow.list is empty
set up task box file - /home/melezhik/.sparrowdo//opt/sparky-sparrowdo/project1/task-box.json - OK

found ini file: /home/melezhik/sparrowdo.ini
sparrow root: [/opt/sparky-sparrowdo/project1]
/opt/sparky-sparrowdo/project1/sparrow.index updated OK from https://sparrowhub.org
sparrow root: [/opt/sparky-sparrowdo/project1]
public@bash is uptodate (0.1.8)
running task box from /opt/sparky-sparrowdo/project1/sparrow-cache/task-box.json ... 
@ runs bash command
2018-10-19 19:25:01 : [task] run bash: echo Hello! [path] /modules/bash-command/ [params] envvars:
Hello! My name is Sparky!
ok	scenario succeeded
STATUS	SUCCEED
``` 


So, as we can see task is successful completed, so far we run task on localhost, but we are not limited. Sparkly is also capable ran tasks remotely:



## Over ssh

```
$ nano sparky.yaml

sparrowdo:
  host: '192.168.0.1'
  ssh_private_key: /path/to/ssh_private/key.pem
  ssh_user: sparky
```

## On Docker 

```
$ nano sparky.yaml

sparrowdo:
  docker: 'black_horse'
```


# What I've not said:


Just highlights of that's not been cover through this post


* [Sparky plugins](https://github.com/melezhik/sparky#sparky-plugins) to extend basic functionality ( sending notification email upon tasks finish and so on )


* [Configuring](https://github.com/melezhik/sparky#configure-sparky-workers) Sparky tasks 

* Managing old reports

* Disabling tasks

* Using [mysql/postgresql](https://github.com/melezhik/sparky#running-under-other-databases-engines-mysql-postgresql) database engines

* Debugging Sparky tasks using [task runner](https://github.com/melezhik/sparky#command-line-client).

* [Docker image](https://github.com/melezhik/sparky-docker) with Sparky ready to use


As always your comments and feedback is highly appreciated.