Stacking Serverless Data Structures for a Pomodoro App on AWS
Tags: aws, nextjs, typescript, dynamodb
Published: 2023-04-20 03:19:18

#BuiltWithAmplify #BuiltWithAppSync

Hello friends! Welcome back to another blog post on the Tech Stack Playbook post, your guide to apps, software, and tech (but in a fun way I promise)!

In this post and video linked below, I'm going to discuss the Pomodoro app I built for AWS' AppSync x Amplify Hackathon, as well as some of the key learnings I found along the way building this full-stack application. One of the coolest elements of this project for me, was the data structure manipulations needed to create a pomodoro stream of events chain together, and then the lift of making my local stack example work in the cloud across many users. I certainly learned a lot from this project, so I hope you get a lot out of this blog post and video too! Without further ado, let's dive in 🚀

## YouTube Video + Demo:

{% youtube ynlBIRSdBl8 %}

## Application page:

Link: https://prod.d1hzn1hof27la1.amplifyapp.com/

# Step 1: Quick Design Mockup (aka map it out)
Something I try to get into the habit of is designing at the most basic level what it is I want to build. It's easy to let code dictate the design, but I think pausing to envision the outcome you want first helps you (1.) arrive at the destination more quickly, and (2.) ensure you are working towards the outcome clearly and efficiently.

This is certainly not a masterpiece, but it helped me start to think about the data structures that would be involved and what I would need:


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/6l7w83ydd0y5wr786gyx.png)




# Step 2: Architect the Tech Stack
This is another important preliminary step that is needed to ensure a smooth outcome for your project or software. When determining the tools you will use, there are about 3 important criteria to consider:
(1.) **bandwidth** - how much time, how many people, how quickly?
(2.) **expertise** - how much do I know, how much needed to learn?
(3.) **flow** - how can this be done easily, efficiently, correctly?

The below are the technologies I used to build this application. You don't necessarily need to use all of them, but I have a strong preference for using Next.js/TypeScript for it's excellent server side rendering (SSR) functionalities, lightning fast build/render time, and it's integration with Styled Components for building customized front-ends. For AWS, the AppSync, DynamoDB, Lambda, S3, Amplify stack is one I am particularly partial for. By using the Amplify CLI, we can coordinate all of our AWS resources for our CloudFormation stack and make sure everything is packaged up cleanly before we send to the cloud and eventually our users.

Here is a list of the technologies I used to build this app:

### 📲 Front-End:
- **Next.js:** modern app development with Server Side Rendering, dynamic routing, and more
- **TypeScript:** to enforce type safety throughout functions and API requests
- **Styled Components:** for fast component development and scaling UIs
- **Material UI:** React component library

### ⚙️ Back-End:
- **🚦 API: GraphQL <> AWS AppSync:** managed API service and GraphQL transformer based on written schema
- **🗃️ DATABASE: Amazon DynamoDB:** lightning fast NoSQL database optimized for web apps and fast READS/WRITES
- **💻 HOSTING: Amplify Hosting:** managed service optimized for Next.js v13's features and deployment requirements
- **💾 STORAGE: Amazon S3** highly inexpensive and highly scalable storage solution for images and app hosting
- **🔐 AUTH: Amazon Cognito:** managed authentication service that neatly wraps into fine-grained-access-controls around the API



# Step 3: Figuring out numeric sort querying using GraphQL of NoSQL DynamoDB data
This was probably the hardest step for me in this project because at the highest level, when you are running a data stream with chaining, you affect one item and it means you must alter the other elements in the chain accordingly.

TL;DR: you edit one item, so you must edit the others accordingly.

For example, consider the following where we will move an item from the 3rd position at index[2] in the stack to the 1st position at index[0] to re-order the stack and everything pushes down.

```markdown
ORIGINAL     —>   FINAL
[ITEM#1]          [ITEM#3]
[ITEM#2]          [ITEM#1]
[ITEM#3]          [ITEM#2]
[ITEM#4]          [ITEM#4]
```

This made perfect sense on paper, but the implementation of this as stack manipulation got even more confusing when I tried to model an abridged NoSQL data structure to emulate how DynamoDB would be interpreting this.

I would be using AppSync to query the Pomodoro items so that I could do a LIST of those items for each user based on the `order` variable. I figured it would be wise to create a separate `order` integer value in my schema so that I could track a value that would be mutating regularly as the user drag-and-dropped their tasks in their READY stack.


When new items get created, their order would increment by +1 relative to the last item index's order. However, as items get moved around, their order will be updated while the ID of course stays the same. Something like the below was what I needed to solve for:

```markdown
      ORIGINAL         —>             FINAL
[id: 1111; order: 1]          [id: 3333; order: 1] 
[id: 2222; order: 2]          [id: 1111; order: 2] 
[id: 3333; order: 3]          [id: 2222; order: 3]
[id: 4444; order: 4]          [id: 4444; order: 4]
```

In the above example, we have unique a `id` for each project, but the order is changing as `id:3333` goes from `order:3` to `order:1` meanwhile the other 3 items in the stack all have updated order values as well.

This was the big data structures and algorithms conundrum I had to figure out, specifically how AppSync could process these batch updates at once to update many items in DynamoDB at once. 

### The solution:

I tested out a number of different implementations and ended up going with a Global Secondary Index (GSI) on a value `orderList` that takes in a String value and has as its sortKeyField `order` as a integer value. 

```javascript
...
id: ID!
title: String!
order: Int
  // ❌ This GSI didn't work
  @index(
    name: "ItemsByUserID"
    queryField: "listItemsByOrder"
    sortKeyFields: ["timestamp"]
  )
orderList: String @default(value: "CURRENT")
  // ✅ This GSI worked!
  @index(
      name: "OrderItemsByUserID"
      queryField: "listOrderItemsByUserID"
      sortKeyFields: ["order"]
  )
...
```

I found that if all of the DynamoDB objects maintain this structure of an `order` number value and an `orderList` string  value, I could do a list query that I named `listOrderItemsByUserID` where I find all of the items that have `orderList === "CURRENT"` and then sort it in ascending order, this then allowed me to list the items in incrementing order numerically even though I was doing a query off of a string value. 

This query looks like this:

```javascript
query MyQuery {
  listOrderItemsByUserID(
    orderList: "CURRENT", 
    sortDirection: ASC
    ) {
    items {
      id
      order
      title
    }
  }
}
```



# Step 4: Stacking/Re-Stacking the Pomodoro Data Structure

Once I figured out that we could use that type of query pattern to get the data listing correctly with the numeric sort, I needed to figure out how to serve `order` changes from the client to AWS.

I was using a JSON test data structure, which was considerably easier to test and troubleshoot with, but I was not familiar with how to do batch-writing to DynamoDB where mutating one item  in the stack spikes mutations of all the other items in the stack and then this is automatically served to the cloud and saved in real time. I was a bit skeptical about this type of set-up given I had realized I've mainly worked on saving items individually or in a queue, rather than a for loop to to batch through client-side updates. 

This is why this project turned into a very rich learning lesson because I was able to find out a way that you can use AppSync to serve updates to DynamoDB that almost feels like a websocket but significantly less bandwith since we are just doing update mutations rather than instantiating a whole websocket to serve the updates.

I learned about this TypeScript method for batch iterating with these components:
- (1.) **A `for...of` loop:** to iterate through the items in the array. We want to run a function across many different items for AppSync to communicate to DynamoDB.
- (2.) **The `.entries()` method:** ** for spotlighting the various items in the stack. We call this method over the `reorderedItems` array so that we can use this method to get back an `IterableIterator` of `[index, element]` for all of the items in the stack, which in our data model is `[number, ItemDDB]`. 
- (3.) **`Tuple Desctructuring`:** We need to assign new values to the `[index, element]` base structure of the `IterableIterator` into variables, so this is a method that will let us make and track those changes.

```tsx
async function updateItemsOrder(reorderedItems: ItemDDB[]) {
        for (const [index, item] of reorderedItems.entries()) {
          try {
            const input = {
              id: item.id,
              order: index + 1,
            };
            await API.graphql(graphqlOperation(updatePomodoroItem, { input }));
          } catch (error) {
            console.error('Error updating item order:', error);
          }
        }
      }
```

⚠️ Important note with TypeScript: This `IterableIterator` is an advanced iteration feature specific to ES6 (ECMAScript 2015), and not usable or callable in previous ES versions. TS will default to using ES5, which then means that it won't support the iteration we want to use, so we need to do one of two things:
- 1.) Switch to ES6 in our `tsconfig.json` file to change `"target": "es2015",` under the `compilerOptions` object, or
- 2.) Add the `--downlevelIteration` compiler flag by adding `"downlevelIteration": true,` to the `compilerOptions` object.


# Step 5: Hard-Typing with GraphQL and TypeScript

There are numerous reasons for using TypeScript in an enterprise applications, from ensuring error-free and bug-free build code when it goes to production, as well as a consistent and reliable framework for how your data structures should be used or get called in your application.

For this hackathon project, I wanted to practice typing my functions, data structures, and API calls. It was either going to be a great learning experience, or a horrible decision that I would regret forever. Even though this was a hackathon project, we must either go big or go home, as they say. 

<img width="100%" style="width:100%" src="https://media.giphy.com/media/l4FGvN3n3IQi4xZNS/giphy.gif">

## Type Guards
When we are working with AppSync, we will be making requests to get either one item (GET), or many items (LIST), so we will need to ensure we have an interface that can cross-check the data object and make sure what you expect to get is really what you get in your call.

An example of this is my `getItemsFromDDB` async function which queries my global secondary index for my schema, passing in the `orderList` string variable, a request to get the data in ascending (ASC) order, and a filter around the status, as I only want the items that are non-COMPLETED, aka "RUNNING" in my case.

Then I want to make sure that I check against an interface, which I named `isGraphQLResultForGetItemsFromDDB` which accepts the `fetchedItems` I receive and make sure that it matches up with this structure:

```tsx 
interface ItemDDB {
  id: string;
  title: string;
  status: string;
  timeBlock: string;
  minutesLeft: string;
  color: string;
  colorDefault: string;
  itemStatus: string;
  order: number;
}

function isGraphQLResultForGetItemsFromDDB(fetchedItems: any
  ): fetchedItems is GraphQLResult<{ listOrderItemsByUserID: { items: ItemDDB[] } }> {
    return fetchedItems.data && fetchedItems.data.listOrderItemsByUserID && fetchedItems.data.listOrderItemsByUserID.items;
}
```

The whole function, useEffect hook, and type guards looks like this:

```tsx
  const getItemsFromDDB = async () => {
    try {
      const fetchedItems = await API.graphql({
        variables: {
          orderList: "CURRENT", 
          sortDirection: "ASC",
          filter: {
            status: {
              eq: "RUNNING"
          }}
        },
        query: listOrderItemsByUserID,
        authMode: "AMAZON_COGNITO_USER_POOLS"
      })

      if (!isGraphQLResultForGetItemsFromDDB(fetchedItems)) {
        throw new Error('Unexpected response from API');
      }

      if (!fetchedItems.data) {
        throw new Error('No data found');
      }

      const returnedItems = fetchedItems.data.listOrderItemsByUserID.items;
      console.log(returnedItems);
      setItems(returnedItems);

    } catch (error) {
        console.log('Error running getItemsFromDDB:', error)
    }
  }

  useEffect(() => {
    getItemsFromDDB();
  }, [])
```

# Step 6: Shipping Your Project Live
Unsurprisingly enough, Amplify's managed hosting prevents you from pushing error-ridden code into production (I guess this is for better and not worse). What is good about a thorough review like this is that AWS will not push or update just-pushed changes if there are type errors or anything that causes breaks in the code deployment pipeline. 

I thinks this is a great benefit to make sure that you do not push buggy or error-prone code to your users at any given time. This is a great benefit for a clean CI/CD pipeline.

Also, Amplify's way that it seamlessly integrates Server Side Rendering out of the box for deploys is phenomenal. 

Further, as you work on your AppSync schema, you can configure and push your code for the back-end outside of the front-end. This decoupling allows you to start branching out beyond a monorepo and start turning Lambdas (this project has a couple lambda functions updating the public data table based on lambda triggers), Authentication, Databases, and storage into microservices that can be worked on and streamed to the front-end via APIs, Amplify's routing service, and AWS service identifiers in AWS Amplify's `aws-exports.js` file with an authorized IAM role.


Did you learn something new about AWS and the cloud? 💭

Let me know in the comments below! ⬇️

## Subscribe to the Tech Stack Playbook on YouTube:

{% youtube ynlBIRSdBl8 %}

<iframe width="560" height="315" src="https://www.youtube.com/embed/ynlBIRSdBl8" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>

Let me know if you found this post helpful! And if you haven't yet, make sure to check out these free resources below:

- <b>Sign up for my newsletter: <a href="https://brianhhough.mykajabi.com/newsletter-signup" target="_blank" rel="noopener noreferrer">Email List</a></b>
- <b>Follow my Instagram for more: <a href="https://instagram.com/brianhhough" target="_blank" rel="noopener noreferrer">@BrianHHough</a></b>
- <b>Watch my latest <a href="https://youtube.com/brianhhough" target="_blank" rel="noopener noreferrer">YouTube video for more</a></b>
- <b>Listen to my Podcast on <a href="https://podcasts.apple.com/us/podcast/tech-stack-playbook/id1553712082" target="_blank" rel="noopener noreferrer">Apple Podcasts</a> and <a href="https://open.spotify.com/show/01bHbabSyBmPfbc3ORMZa9?si=av_16aPbQEqL_bVu3RnO0g" target="_blank" rel="noopener noreferrer">Spotify</a></b>