Scaling for Success: Uncovering the Hidden Challenge in Payment App Development
Tags: microservices, java, kubernetes, architecture
Published: 2023-11-15 12:41:10

I'd like to share the story of our audit of a fintech app and how we discovered and solved the scalability problem that could have ruined essential metrics if left unaddressed.
 
I’m a Solution Architect at Sibedge, a full-cycle custom software development company. I’ve worked for over 15 years in analytics, software development, and support for payment processing systems. Among the challenges we've encountered on our projects, one recent project particularly stands out, as it sheds light on a critical issue that often goes unnoticed until it's too late.


## The Problem: Lack of Auto-scaling Capabilities


At Sibedge, we're committed to delivering reliable and efficient software solutions to our clients, and our payment app project, codenamed FinApp, was no exception. During our thorough audit of the application, we were pleased to find that many aspects of its architecture were well-designed, ensuring robust security and high availability.
However, as we delved more deeply, a significant concern came to light
— **the lack of auto-scaling capabilities.**
 
Auto-scaling may seem like a minor detail, but it plays a **crucial role in maintaining the reliability and performance of a system**, especially in the face of unforeseen challenges. Our team was conducting stress testing when we realized that without this essential feature, the FinApp system was vulnerable to performance degradation during heavy spikes of incoming requests, like during sales and promotions, bill due days, or stock market activities. The result of stress testing showed that as the number of requests grows, the number of returned errors grows as well — the transactions are not completed or fail altogether.
 

## Consequences of the Problem

 
The consequences of not having auto-scaling capabilities could have been severe. When a payment app like FinApp faces a sudden surge in traffic due to unexpected high-demand situations, the results can include:
 
**1. Downtime:** The system might struggle to handle the influx of requests, leading to prolonged downtime and frustrated, dissatisfied users.
**2. Performance Degradation:** Even if the system doesn't completely crash, it can experience severe performance degradation, resulting in slow response times and a subpar user experience. Or, even more frustrating for users, the response is so slow that it goes beyond the timeout, and the user receives no response from the system at all.
**3. Resource Inefficiency:** Without auto-scaling, there might be more simultaneous instances of the app launching even during periods of low demand, leading to unnecessary costs.
**4. Manual Intervention:** Maintaining performance in such scenarios would require manual intervention, which is unnecessarily time-consuming and prone to human errors. We might need to hire a dedicated full-time DevOps team for the situational scaling control alone.


## Discovering the Problem

Our discovery of the lack of auto-scaling capabilities was not immediate. It came to light during load testing and IT audits, which we consider fundamental steps in our development process. As we simulated various scenarios, including heavy traffic, we noticed FinApp struggled to adapt to the increased load effectively.
We imitated peak loads the app might experience during Christmas, 11/11 etc. when TPS (Transactions Per Second) can go up to 4-5k. On regular days, TPS is rarely over 1.5k, so we needed to scale the services flexibly.
 

## The Solution: Kubernetes and Horizontal Pod Autoscaler (HPA)
 
Manual scaling was not an option — it required having a 24/7 operation team that would react to any spikes or other load anomalies to scale up/down the system in time. We could not rely on code refactoring either, so we needed an external orchestrator to scale only during peaks.
 
We could also look into VPA (Vertical Pod Scaling), but let’s compare the pros and cons of both VPA and HPA:
 
**VPA:**
+ a more straightforward and easier way to scale;
- not optimal in terms of resource consumption;
- VPA can't help if the app can't consume it. The FinApp design had limits and an inability to process more than 20 requests at the time (and we couldn't rewrite it).
 
**HPA:**
+ more appropriate resource consumption;
- requires an app to be scalable (stateless), but it is in our case.
 
After considering these factors, we proposed leveraging the existing Kubernetes cluster by **implementing the automatic scaling of microservices** using Kubernetes with its Horizontal Pod Autoscaler (HPA) feature. It allowed us to automate the scaling process based on actual CPU and RAM load.
 
By adopting automatic scaling with Kubernetes and HPA, the FinApp system achieved high scalability, ensuring efficient handling of incoming requests without manual intervention. Having a pod with a limit of 2 GB / 500 mCPU, we realized that the scaleUp policy should occur when the average CPU utilization is above 50% on a pod to raise new services, but not more than 15, due to system resource limitations. We found that the optimal scaleDown strategy is to extinguish at 10% power for 20 minutes without critical CPU utilization.
 
This dynamic adjustment of resources based on demand optimized performance and effectively managed costs. It transformed the application into a **responsive** and **cost-effective** microservices environment capable of handling 3-5k TPS on peaks and automatically scaling down on normal days when low TPS is enough.
 

## Conclusion
 
In today's fast-paced digital world, auto-scaling is not a luxury but a necessity, ensuring your system can thrive under the most demanding conditions.
 
As we continue our journey of crafting reliable software solutions, **we invite you to join us in our mission** to tackle typical microservices architecture issues in Payment Solutions.
 
**Sign up for our free webinar**, where we will explore other critical microservices architecture design mistakes and how to avoid and mitigate them.
[**Register and read more about it!**](https://events.sibedge.com/microservicesissues_webinar?utm_source=dev)