Dealing With OutOfMemoryError in Spring Boot
Tags: programming, softwaredevelopment, softwareengineering, springboot
Published: 2024-01-19 20:33:49

Working as a freelancer, [I spend my time on different projects](https://writech.run/blog/reinventing-kanban-with-notion-2586bb187638). During the last one, like many others, I had to design and build a CRUD application. My usual approach is to use the multi-layered architecture described [here](https://writech.run/blog/designing-a-multi-layered-architecture-for-building-restful-web-services-with-spring-boot-and-a12ef85b77c9).

I've never had problems, but this time it was different. Due to performance issues, I had to push my back-end to the limit. **Extremely complex queries, multiple levels of parallelization, bigger data, and more requests than usual**. Although I don't call myself a Spring Boot expert, I'm pretty confident about how to use it. However, **every 3 or 4 days, an OutOfMemoryError occurred**.

This is the story of how I fixed it, improving performance as a side effect, and what this taught me.

## Specs and Architecture

The [Spring Boot](https://spring.io/projects/spring-boot) web application is running on a [Docker](https://www.docker.com/) container handled by [Dokku](http://dokku.viewdocs.io/dokku/). I have been using this deployment system for years and nothing wrong has ever happened, so I am pretty sure my problem does not stand here.

The **4GB 2 CPU** [**VPS**](https://it.wikipedia.org/wiki/Virtual_private_server) I am using is enough for such a project, especially considering that we do not expect so many users.

My [`Procfile`](https://devcenter.heroku.com/articles/procfile) is very simple and scaled on available resources:

```shell
web: java -Xms1024M -Xmx4096M -jar build/libs/my-app-1.0.jar
```

- 1GB as initial heap size \[`-Xms1024M` \]

- 4GB as maximum heap size \[`-Xmx4096M` \]

The database is a [MySql](https://www.mysql.com/) server running on another VPS, so they are not sharing the same resources.

## The Recurring Error

The back-end runs flawlessly for about two weeks. Day by day, data becomes bigger, and users making concurrent requests increase. **During the third week, I get the first** [**`OutOfMemoryError`**](https://docs.oracle.com/javase/8/docs/api/java/lang/OutOfMemoryError.html). Logs are pretty clear: **the VPS failed while trying to create a new thread, running out of memory**. I immediately ask for advice from [Dario Pellegrini](https://medium.com/u/6499c7edd7fb), a colleague and developer with 5+ years of experience. Together we decide to **double the RAM of the VPS and the maximum heap size limit consequently**.

Problem fixed, right?

Nope!

3 days later, **the same `OutOfMemoryError` exception strikes again!** This time I want to tackle the problem for real and look for a real solution.

These are my attempts to fix it…

### 1\. Trying a Practical Solution

Running some tests I find out that **a query returning 20k rows can easily become the bottleneck of an application**. This is especially true if the [`@Entity` mapping class](https://writech.run/blog/defining-jpa-hibernate-entities-in-kotlin-1ff8ee470805/) consists of more than 130 attributes (just like mine). Instead of retrieving 10/20k rows each time, mapping them into Hibernate objects, iterate through them to retrieve a single computed value, I replace it all with a very advanced query. By defining a custom mapping class and writing a highly optimized query, **I am able to achieve the same result with a much more efficient single-row query**. With this (not so) simple trick, **I increase the performance by 70/80% and reduce significantly the heap size**.

Not to brag, but this time I'm pretty sure I've solved the problem…

Not yet…

### 2\. Looking for Esoteric Reasons

Since the story is still going on, I'm not proud to say that it didn't work out. **6 days later my back-end fails for the same `OutOfMemoryError`**.

I decide to take it seriously this time. I look for someone with a similar issue online and come across some interesting, but **somewhat esoteric**, topics.

Firstly, a couple of APIs are really complex and require hundreds of queries. Then, the [query plan cache provided by Hibernate](https://www.baeldung.com/hibernate-query-plan-cache) may be the culprit. **By default, the maximum number of entries in the plan cache is** [**2048**](https://docs.jboss.org/hibernate/orm/4.1/javadocs/org/hibernate/cfg/AvailableSettings.html#QUERY_PLAN_CACHE_MAX_SIZE)**.** An `HQLQueryPlan` object occupies approximately [3MB](https://medium.com/@truongnhukhang1993/the-way-i-detect-out-of-memory-error-in-a-spring-boot-spring-data-web-application-d9648effc928). ~3 \* 2048 = ~6GB, and the heap size is limited to 4GB…

Finally! That must be the cause!

The solution is simple: **decrease the query plan cache size** by setting the following properties:

- `spring.jpa.properties.hibernate.query.plan_cache_max_size`: controls the maximum number of entries in the plan cache (defaults to **2048**)

- `spring.jpa.properties.hibernate.query.plan_parameter_metadata_max_size`: manages the number of `ParameterMetadata` instances in the cache (defaults to **128**)

I set them to **1024** and **64** respectively.

Secondly, I find out that **`IN`** **queries with a variable number of parameters are a bit tricky for the Hibernate query plan cache** (further reading [here](https://stackoverflow.com/a/40017486) and [here](https://vladmihalcea.com/improve-statement-caching-efficiency-in-clause-parameter-padding/)).

The solution is easy again. You can improve the statement caching efficiency with [`IN`](https://vladmihalcea.com/improve-statement-caching-efficiency-in-clause-parameter-padding/) [**clause parameter padding**](https://vladmihalcea.com/improve-statement-caching-efficiency-in-clause-parameter-padding/) by setting the following property to _true_:

```shell
spring.jpa.properties.hibernate.query.in_clause_parameter_padding
```

Thirdly, my application is multi-thread. Since the error seems to occur while it tries to create a new thread, why not **reduce thread stack size from 1024KB to 512KB** by adding [`-Xss512k`](https://docs.oracle.com/cd/E13150_01/jrockit_jvm/jrockit/jrdocs/refman/optionX.html) to my `Procfile`?

Each of these properties should **decrease the amount of heap used and therefore fix the `OutOfMemoryError` problem**, once and for all. Right? Nope!

### 3\. Surrendering to Reality

Last but not least, I decide to do what I should have done immediately: review my code! The [JVM Garbage Collector](https://www.baeldung.com/jvm-garbage-collectors) gathers all the unreachable objects to try to delete them. In the case of too many unreachable objects, the heap will fill up before the GC intervenes. In this case, **an `OutOfMemoryError` exception is thrown as soon as the application tries to allocate a new object**. Reviewing my code I find nothing suspicious.

My application is multi-thread and I used [thread pools](https://www.baeldung.com/thread-pool-java-and-guava) more than once, especially [`ForkJoinPool`](https://www.baeldung.com/thread-pool-java-and-guava). I have not used it for ages, so the problem could be there. **I started looking online for well-known memory leaks** and during my researches, I come across this interesting [GitHub page](https://github.com/jacek-rzrz/ForkJoinPool-thread-leak). I immediately realize what a beginner mistake I made. **I forgot to shut down each of the** **`ForkJoinPool`** **objects I used**!

Oops!

## Conclusion

This experience taught me how complex problems can have very simple solutions. Instead of stubbornly looking for any esoteric reason or insufficient resources, you should consider trivial errors first. I hope this story helps you not to make the same mistake I did.
  
***
_The post "[Dealing With OutOfMemoryError in Spring Boot](https://writech.run/blog/dealing-with-outofmemoryerror-in-spring-boot-11f2061063a1/)" appeared first on [Writech](https://writech.run)._