Easily process long-running jobs with JobRunr
Tags: java, distributedsystems
Published: 2020-04-24 08:22:25

In this tutorial, we will be working for the fictional company Acme Corp and we need to generate the salary slips for all of Acme Corp's employees.

> TLDR; you can find the complete project on our Github repository: https://github.com/jobrunr/example-salary-slip

![JobRunr](https://www.jobrunr.io/content/images/2020/04/IMG_2293-3.JPG)

To do so, we will be using 3 open-source components:
- [JobRunr](https://www.jobrunr.io): JobRunr allows to easily schedule and process background jobs using Java 8 lambda's. It is backed by persistent storage and can process jobs in a parallel and distributed manner. Thanks to the built-in dashboard we have an in-depth overview into all our background jobs.
- [Spring Data Jpa](https://docs.spring.io/spring-data/jpa/docs/current/reference/html): If you want to easily access data in a relational database, Spring Data Jpa is here to help. You can create repositories using nothing more than a simple interface
- [Docx-Stamper](https://github.com/thombergs/docx-stamper): Docx-Stamper allows to easily generate Word (.docx) documents backed by templates

<!-- To add some whitespace -->
&nbsp; 

# Architecture
During this tutorial, we will generate the weekly salary slip of all of Acme Corp's employees and email it to them. How? Well, by
* creating a recurring job using JobRunr that will run every week - it will get all of Acme Corp's employees using Spring Data Jpa and for each of these employees schedule a new background job to create the salary slip
* each of these background jobs will fetch the Employee and 
  * consume a `TimeClockService` which gives the amount of hours an employee worked for the given week.
  * generate a salary slip document using a `DocumentGenerationService` which will contain the name of the employee and the amount of hour he or she worked. The salary slip document is generated from a Word template and converted to a PDF file.
  * send an email to the employee with his salary slip using an `EmailService`.

<!-- To add some whitespace -->
&nbsp; 

# Let's gets started!
In this tutorial we omit all Java imports for brevity - to find them, just visit the example project on https://github.com/jobrunr/example-salary-slip

For building this salary slip service, we use gradle and our build.gradle file is as follows:
```
plugins {
    id 'java'
    id 'idea'
    id 'org.springframework.boot' version '2.2.2.RELEASE'
    id 'io.spring.dependency-management' version '1.0.8.RELEASE'
}

group 'org.paycheck'
version '1.0-SNAPSHOT'

repositories {
    mavenCentral()
    jcenter()
    mavenLocal()
}

configurations.all {
    exclude group: 'org.slf4j', module: 'slf4j-log4j12'
}

dependencies {
    implementation 'org.jobrunr:jobrunr:0.9.2'
    implementation 'org.springframework.boot:spring-boot-starter-data-jpa'
    implementation 'org.springframework.boot:spring-boot-starter-mail'
    implementation 'com.fasterxml.jackson.core:jackson-databind'
    implementation 'com.fasterxml.jackson.datatype:jackson-datatype-jsr310'
    implementation 'org.wickedsource.docx-stamper:docx-stamper:1.4.0'
    implementation 'org.docx4j:docx4j-core:8.1.6'
    implementation 'org.docx4j:docx4j-JAXB-ReferenceImpl:11.1.3'
    implementation 'org.docx4j:docx4j-export-fo:8.1.6'


    implementation 'com.github.javafaker:javafaker:1.0.2'
    implementation 'com.h2database:h2'

    testImplementation 'org.junit.jupiter:junit-jupiter:5.6.1'
    testImplementation 'org.awaitility:awaitility:4.0.2'
}

test {
    useJUnitPlatform()
}
```

<!-- To add some whitespace -->
&nbsp; 

### Employee entity
Since we need to create salary slips for all employees let us start with the `Employee` class - it is a simple Entity with some fields like `firstName`, `lastName` and `email`.

```java
package org.jobrunr.example.employee;

@Entity
public class Employee {

    @Id
    @GeneratedValue(strategy = GenerationType.AUTO)
    private Long id;
    private String firstName;
    private String lastName;
    private String email;

    protected Employee() {
    }

    public Employee(String firstName, String lastName, String email) {
        this(null, firstName, lastName, email);
    }

    public Employee(Long id, String firstName, String lastName, String email) {
        this.id = id;
        this.firstName = firstName;
        this.lastName = lastName;
        this.email = email;
    }

    @Override
    public String toString() {
        return String.format(
                "Employee[id=%d, firstName='%s', lastName='%s']",
                id, firstName, lastName);
    }

    public Long getId() {
        return id;
    }

    public String getFirstName() {
        return firstName;
    }

    public String getLastName() {
        return lastName;
    }

    public String getEmail() {
        return email;
    }
}
```

<!-- To add some whitespace -->
&nbsp; 

### EmployeeRepository

The `EmployeeRepository` extends the Spring Data `CrudRepository` and adds an extra method to fetch all the id's of the Employees

```java
package org.jobrunr.example.employee;

public interface EmployeeRepository extends CrudRepository<Employee, Long> {

    @Query("select e.id from Employee e")
    Stream<Long> getAllEmployeeIds();

}
```

<!-- To add some whitespace -->
&nbsp; 

### WorkWeek domain class
Since the salary slip is generated once per week, we need a class representing the amount of time an employee has worked that week - the `WorkWeek` class. It has some extra fields like the weekNbr and a from and to date which we will use for our generated salary slip document.

```java
package org.jobrunr.example.timeclock;

public class WorkWeek {

    private final int weekNbr;
    private final BigDecimal workHoursMonday;
    private final BigDecimal workHoursTuesday;
    private final BigDecimal workHoursWednesday;
    private final BigDecimal workHoursThursday;
    private final BigDecimal workHoursFriday;
    private final LocalDate from;
    private final LocalDate to;

    public WorkWeek(BigDecimal workHoursMonday, BigDecimal workHoursTuesday, BigDecimal workHoursWednesday, BigDecimal workHoursThursday, BigDecimal workHoursFriday) {
        this.workHoursMonday = workHoursMonday;
        this.workHoursTuesday = workHoursTuesday;
        this.workHoursWednesday = workHoursWednesday;
        this.workHoursThursday = workHoursThursday;
        this.workHoursFriday = workHoursFriday;
        WeekFields weekFields = WeekFields.of(Locale.getDefault());
        weekNbr = now().get(weekFields.weekOfWeekBasedYear());
        this.from = now().with(TemporalAdjusters.previous(DayOfWeek.MONDAY));
        this.to = now().with(TemporalAdjusters.nextOrSame(DayOfWeek.SUNDAY));
    }

    public BigDecimal getWorkHoursMonday() {
        return workHoursMonday;
    }

    public BigDecimal getWorkHoursTuesday() {
        return workHoursTuesday;
    }

    public BigDecimal getWorkHoursWednesday() {
        return workHoursWednesday;
    }

    public BigDecimal getWorkHoursThursday() {
        return workHoursThursday;
    }

    public BigDecimal getWorkHoursFriday() {
        return workHoursFriday;
    }

    public int getWeekNbr() {
        return weekNbr;
    }

    public LocalDate getFrom() {
        return from;
    }

    public LocalDate getTo() {
        return to;
    }

    public BigDecimal getTotal() {
        return workHoursMonday
                .add(workHoursTuesday)
                .add(workHoursWednesday)
                .add(workHoursThursday)
                .add(workHoursFriday);
    }
}
```

<!-- To add some whitespace -->
&nbsp; 

### TimeClockService
To get a `WorkWeek` class for a certain employee, we create a `TimeClockService` which is a Spring Component. As we don't want to make this tutorial overly complex, here we use a stub which generates some random data. In a real-world application this service would make a REST or SOAP request to another microservice. 

```java
package org.jobrunr.example.timeclock;

@Component
public class TimeClockService {

    public WorkWeek getWorkWeekForEmployee(Long employeeId) {
        try {
            //simulate a long-during call
            Thread.sleep(ThreadLocalRandom.current().nextInt(3, 5 + 1) * 1000);
            return new WorkWeek(
                    BigDecimal.valueOf(getRandomHours()),
                    BigDecimal.valueOf(getRandomHours()),
                    BigDecimal.valueOf(getRandomHours()),
                    BigDecimal.valueOf(getRandomHours()),
                    BigDecimal.valueOf(getRandomHours())
            );
        } catch (InterruptedException e) {
            Thread.currentThread().interrupt();
            throw new RuntimeException(e);
        }
    }

    private int getRandomHours() {
        Random r = new Random();
        return r.nextInt((8 - 5) + 1) + 5;
    }
}
```

<!-- To add some whitespace -->
&nbsp; 

### SalarySlip
We now have all the necessary data to generate our salary slip - except the `SalarySlip` class itself: 

```java
package org.jobrunr.example.paycheck;

public class SalarySlip {

    private final Employee employee;
    private final WorkWeek workWeek;

    public SalarySlip(Employee employee, WorkWeek workWeek) {
        this.employee = employee;
        this.workWeek = workWeek;
    }

    public Employee getEmployee() {
        return employee;
    }

    public WorkWeek getWorkWeek() {
        return workWeek;
    }

    public BigDecimal getTotal() {
        BigDecimal totalPerHour = getTotalPerHour();
        BigDecimal amountOfWorkedHours = getAmountOfWorkedHours();
        return totalPerHour.multiply(amountOfWorkedHours).setScale(2, RoundingMode.HALF_UP);
    }

    private BigDecimal getAmountOfWorkedHours() {
        return workWeek.getTotal();
    }

    private BigDecimal getTotalPerHour() {
        return BigDecimal.valueOf(21.50);
    }
}
```

The `SalarySlip` class contains all the data necessary to generate a salary slip, calculate the total amount and will be used by the `DocumentGenerationService` to generate the salary slips as PDF documents.

<!-- To add some whitespace -->
&nbsp; 

### DocumentGenerationService

The `DocumentGenerationService` is also a Spring Component and has the responsibility to generate the actual salary slip documents based on a word template. The [word template](https://github.com/jobrunr/example-salary-slip/blob/master/src/main/resources/templates/salary-slip-template.docx) has a lot of placeholders, like ${employee.firstName}, ${employee.lastName} and ${workWeek.workHoursMonday.setScale(2)} that will be replaced by DocxStamper using the given context object - in our case a SalarySlip object. Finally, the Word document with all fields filled in, is converted to a PDF document.

```java
package org.jobrunr.example.paycheck;

@Component
public class DocumentGenerationService {

    public void generateDocument(Path wordTemplatePath, Path wordOutputPath, Object context) throws IOException, Docx4JException {
        Files.createDirectories(wordOutputPath.getParent().toAbsolutePath());

        try(InputStream template = Files.newInputStream(wordTemplatePath); OutputStream out = Files.newOutputStream(wordOutputPath)) {
            final ByteArrayOutputStream byteArrayOutputStream = new ByteArrayOutputStream();
            final DocxStamper stamper = new DocxStamperConfiguration().setFailOnUnresolvedExpression(true).build();
            stamper.stamp(template, context, byteArrayOutputStream);

            Docx4J.toPDF(WordprocessingMLPackage.load(new ByteArrayInputStream(byteArrayOutputStream.toByteArray())), out);
        }
    }
}
```

<!-- To add some whitespace -->
&nbsp; 

### EmailService

The `EmailService` is again a Spring Component and has the responsibility to email the final salary slip word document to the employee - it uses Spring Boot Starter Email and using a `MimeMessage` created by the `JavaMailSender`. It has a method called `sendSalarySlip` with two arguments - the `Employee` class and the path to the salary slip PDF document for that employee. Using these argument, we can send both a personalized text email and attach the actual salary slip as an attachment.
The `JavaMailSender` is a class provided by Spring Boot Starter Mail and configured using a properties file. It is configured using a properties file you can find [here](https://github.com/jobrunr/example-salary-slip/blob/master/src/main/resources/application.properties)

```java
package org.jobrunr.example.email;

@Component
public class EmailService {

    public JavaMailSender emailSender;

    public EmailService(JavaMailSender emailSender) {
        this.emailSender = emailSender;
    }

    public void sendSalarySlip(Employee employee, Path salarySlipPath) throws MessagingException {
        MimeMessage message = emailSender.createMimeMessage();
        MimeMessageHelper helper = new MimeMessageHelper(message, true);
        helper.setTo(employee.getEmail());
        helper.setSubject("Your weekly salary slip");
        helper.setText(String.format("Dear %s,\n\nhere you can find your weekly salary slip. \n \nThanks again for your hard work,\nAcme corp", employee.getFirstName()));

        FileSystemResource file = new FileSystemResource(salarySlipPath);
        helper.addAttachment("Salary Slip", file);
        emailSender.send(message);
    }
}
```

<!-- To add some whitespace -->
&nbsp; 

### And finally, the SalarySlipService
The SalarySlipService is the last step of the puzzle and wires everything together:
- it has the path to the salary slip word template
- it uses the other components we already created:
  - the `EmployeeRepository` to get all employees from the database
  - the `TimeClockService` to get the amount of hours an employee worked
  - the `DocumentGenerationService` to create a salary slip from the given Word template
  - the `EmailService` to send a personalized email with the salary slip as attachment to the employee

*It also two important public methods:*
#### `generateAndSendSalarySlip`
The method `generateAndSendSalarySlip` uses the employee id to get the actual employee data, generates the salary slip PDF document and sends it via email to the employee. It will be a JobRunr background job and it is called from the method `generateAndSendSalarySlipToAllEmployees`. We annotate it with the Job annotation (this is optional) to have meaningful names in dashboard of JobRunr.

```java
    @Job(name = "Generate and send salary slip to employee %0")
    public void generateAndSendSalarySlip(Long employeeId) throws Exception {
        final Employee employee = getEmployee(employeeId);
        Path salarySlipPath = generateSalarySlip(employee);
        emailService.sendSalarySlip(employee, salarySlipPath);
    }
```

#### `generateAndSendSalarySlipToAllEmployees`
Our main method that will be scheduled each week - it gets a stream of employee ids and using the `BackgroundJob.enqueue` method of JobRunr, we create a background job of the `generateAndSendSalarySlip` method for each employee. 
The document generation fails because there is not enough disk space? Or the `TimeClockService` fails for an employee because the external microservice is down? No worries - as JobRunr is fault-tolerant (it will automatically retry failed jobs with an exponential back-off policy), these failing jobs will be retried 10 times automatically.

```java
    @Transactional(readOnly = true)
    @Job(name = "Generate and send salary slip to all employees")
    public void generateAndSendSalarySlipToAllEmployees() {
        final Stream<Long> allEmployees = employeeRepository.getAllEmployeeIds();
        BackgroundJob.<SalarySlipService, Long>enqueue(allEmployees, (salarySlipService, employeeId) -> salarySlipService.generateAndSendSalarySlip(employeeId));
    }
```

<!-- To add some whitespace -->
&nbsp; 

The complete `SalarySlipService` is as follows:
```java
package org.jobrunr.example.paycheck;

@Component
public class SalarySlipService {

    private static final Path salarySlipTemplatePath = Path.of("src/main/resources/templates/salary-slip-template.docx");

    private final EmployeeRepository employeeRepository;
    private final TimeClockService timeClockService;
    private final DocumentGenerationService documentGenerationService;
    private final EmailService emailService;

    public SalarySlipService(EmployeeRepository employeeRepository, TimeClockService timeClockService, DocumentGenerationService documentGenerationService, EmailService emailService) {
        this.employeeRepository = employeeRepository;
        this.timeClockService = timeClockService;
        this.documentGenerationService = documentGenerationService;
        this.emailService = emailService;
    }

    @Transactional(readOnly = true)
    @Job(name = "Generate and send salary slip to all employees")
    public void generateAndSendSalarySlipToAllEmployees() {
        final Stream<Long> allEmployees = employeeRepository.getAllEmployeeIds();
        BackgroundJob.<SalarySlipService, Long>enqueue(allEmployees, (salarySlipService, employeeId) -> salarySlipService.generateAndSendSalarySlip(employeeId));
    }

    @Job(name = "Generate and send salary slip to employee %0")
    public void generateAndSendSalarySlip(Long employeeId) throws Exception {
        final Employee employee = getEmployee(employeeId);
        Path salarySlipPath = generateSalarySlip(employee);
        emailService.sendSalarySlip(employee, salarySlipPath);
    }

    private Path generateSalarySlip(Employee employee) throws Exception {
        final WorkWeek workWeek = getWorkWeekForEmployee(employee.getId());
        final SalarySlip salarySlip = new SalarySlip(employee, workWeek);
        return generateSalarySlipDocumentUsingTemplate(salarySlip);
    }

    private Path generateSalarySlipDocumentUsingTemplate(SalarySlip salarySlip) throws Exception {
        Path salarySlipPath = Paths.get(System.getProperty("java.io.tmpdir"), String.valueOf(now().getYear()), format("workweek-%d", salarySlip.getWorkWeek().getWeekNbr()), format("salary-slip-employee-%d.docx", salarySlip.getEmployee().getId()));
        documentGenerationService.generateDocument(salarySlipTemplatePath, salarySlipPath, salarySlip);
        return salarySlipPath;
    }

    private WorkWeek getWorkWeekForEmployee(Long employeeId) {
        return timeClockService.getWorkWeekForEmployee(employeeId);
    }

    private Employee getEmployee(Long employeeId) {
        return employeeRepository.findById(employeeId).orElseThrow(() -> new IllegalArgumentException(format("Employee with id '%d' does not exist", employeeId)));
    }
}
```

<!-- To add some whitespace -->
&nbsp; 

### Last but not least - our Spring Boot Application
The Spring Boot Application bootstraps our application and has one important piece of code:

```java
    BackgroundJob.scheduleRecurringly(
        "generate-and-send-salary-slip",
        SalarySlipService::generateAndSendSalarySlipToAllEmployees,
        Cron.weekly(DayOfWeek.SUNDAY, 22)
);
```

This method call ensures that the `generateAndSendSalarySlipToAllEmployees` method of our `SalarySlipService` will be triggered each Sunday at 10pm.
In this SpringBootApplication we create some fake employees, define a `DataSource` (in our case a simple H2 database) and initialize JobRunr using it's fluent-api.

```java
package org.jobrunr.example;

@SpringBootApplication
public class SalarySlipMicroService {

    public static void main(String[] args) {
        SpringApplication.run(SalarySlipMicroService.class, args);
    }

    @Bean
    public CommandLineRunner demo(EmployeeRepository repository) {
        final Faker faker = new Faker();
        return (args) -> {
            for(int i = 0; i < 1000; i++) {
                repository.save(new Employee(faker.name().firstName(), faker.name().lastName(), faker.internet().emailAddress()));
            }

            BackgroundJob.scheduleRecurringly(
                    "generate-and-send-salary-slip",
                    SalarySlipService::generateAndSendSalarySlipToAllEmployees,
                    Cron.weekly(DayOfWeek.SUNDAY, 22)
            );

            Thread.currentThread().join();
        };
    }

    @Bean
    public DataSource dataSource() {
        final JdbcDataSource ds = new JdbcDataSource();
        ds.setURL("jdbc:h2:" + Paths.get(System.getProperty("java.io.tmpdir"), "paycheck"));
        ds.setUser("sa");
        ds.setPassword("sa");
        return ds;
    }

    @Bean
    public JobScheduler initJobRunr(ApplicationContext applicationContext) {
        return JobRunr.configure()
                .useStorageProvider(SqlStorageProviderFactory
                        .using(applicationContext.getBean(DataSource.class)))
                .useJobActivator(applicationContext::getBean)
                .useDefaultBackgroundJobServer()
                .useDashboard()
                .initialize();
    }

}
```

<!-- To add some whitespace -->
&nbsp; 

# Time to use our new application!
Once you start the SalarySlipMicroService application, you can open your browser using the url http://localhost:8000/dashboard/ and navigate to the Recurring jobs tab.

<!-- To add some whitespace -->
&nbsp; 

![Recurring Jobs](https://www.jobrunr.io/content/images/2020/04/tutorial-salary-slip-01-recurring-job.png)

To test it, we trigger it now manually. The job is processed and schedules a new job to create the salary slip for each employee. Within 15 seconds the processing of these jobs start and we will see the generated PDF documents in our tmp folder.

<!-- To add some whitespace -->
&nbsp; 
![Enqueued Jobs](https://www.jobrunr.io/content/images/2020/04/tutorial-salary-slip-02.png)


We can inspect a Job and see them succeed - if it would fail for some reason, they will be automatically retried.

<!-- To add some whitespace -->
&nbsp; 
![Succeeded Job](https://www.jobrunr.io/content/images/2020/04/tutorial-salary-slip-04.png)

## Conclusion:
- JobRunr and Spring Data integrate very well and both are very easy to use. Being able to schedule Java 8 lambda's and have them run in a background process is a really nice feature of JobRunr.
- To convert the Word document to PDF, there is some nasty stuff in the word template (like white text) to have an OK-layout. Docx-Stamper is a great library and depends on Docx4J. Docx4J allows to convert Word documents to PDF but it still requires some work as a couple of hacks were done to get the layout right. The conversion from Word to PDF also takes a lot of time - ideal thus for JobRunr as it is a long-running background job.