Tackling a Tough Backend Challenge: Building a Distributed Task Scheduler
Published: 2024-07-01 16:38:05



Hey there! I'm thrilled to share my recent adventure in backend development with you. As someone who loves diving into complex problems, I decided to take on the task of building a distributed task scheduler app. This project pushed me to my limits but taught me so much along the way.

### The Problem

So, what was the big idea? I wanted to create a distributed task scheduler that could handle both one-time and recurring tasks, making sure each task was executed within 10 seconds of its scheduled time. Sounds simple enough, right? Well, as I soon discovered, it was anything but straightforward.

### The Journey

#### Step 1: Designing the Architecture

The first hurdle was designing the system's architecture. I opted for a microservices approach to keep things flexible and scalable. The key components were:

- **Task API Service**: This would handle all the task-related CRUD operations.
- **Scheduler Service**: Responsible for scheduling tasks accurately.
- **Worker Nodes**: These would actually execute the tasks.
- **Database**: For storing task data and logs.
- **Message Queue**: To keep communication flowing smoothly between services.

#### Step 2: Picking the Right Tools on GCP

I chose Google Cloud Platform (GCP) for this project and selected these services:

- **Google Cloud Pub/Sub**: To manage messaging between the different services.
- **Google Cloud Firestore**: As our database.
- **Google Cloud Functions**: For executing tasks.
- **Google Cloud Scheduler**: To manage the scheduling of recurring tasks.

#### Step 3: Building the Task API

With the architecture set, I dived into coding the Task API using TypeScript and Express.js. This service offered endpoints to create, update, and delete tasks, and it published task info to a Pub/Sub topic.

#### Step 4: Creating the Scheduler Service

This part was a real brain-teaser. The Scheduler Service needed to listen for new tasks, calculate when to run them, and schedule them precisely. For recurring tasks, I used Google Cloud Scheduler, which made managing and triggering recurring tasks straightforward and reliable.

#### Step 5: Setting Up the Worker Nodes

The Worker Nodes were essentially Google Cloud Functions that subscribed to Pub/Sub topics and executed the tasks when their time came. This setup ensured tasks were executed on time.

#### Step 6: Developing the GUI

Finally, I integrated a user-friendly GUI using React. This allowed users to manage tasks easily and see their status and execution logs in real-time.

### The Hiccups

This journey wasn't without its challenges:

- **Timely Execution**: Getting tasks to run within that 10-second window was tricky. I had to fine-tune everything to minimize delays.
- **Error Handling**: Making the system resilient to failures involved implementing robust error handling and retry mechanisms.
- **Scalability**: Ensuring the system could handle a growing number of tasks required careful optimization.

### Wrapping Up

Building this distributed task scheduler was a rollercoaster ride, but it was incredibly rewarding. I learned a ton about distributed systems and cloud services, and it reinforced the importance of planning and testing.

As I gear up for the HNG Internship, I'm excited to continue growing and learning. This program is an amazing opportunity to work on real-world projects and learn from experienced mentors. I can't wait to see where this journey takes me!

If you're curious about the HNG Internship, check out their [website](https://hng.tech/internship). And if you're looking to hire talented developers, head over to [HNG Hire](https://hng.tech/hire).