Clever Caching | Stock Analysis app using Netlify's fine-grained cache control
Tags: netlifychallenge, devchallenge, webdev, javascript
Published: 2024-05-12 12:53:47

*This is a submission for the [Netlify Dynamic Site Challenge](https://dev.to/challenges/netlify): Clever Caching*


## What I Built
<!-- Share an overview about your project. -->
I have build a Stock Analysis tool which shows price history, financial data and related News articles for any NYSE listed stock.
[Github Link](https://github.com/ashiqsultan/market-lens)
### Cache Mechanisms used in the Project
- **`Netlify-Vary`** headers
- **`Netlify-CDN-Cache-Control`** headers
- SWR (Stale while revalidate)  

### On-demand Cache Invalidation mechanism using
- **`purgeCache()`** helper function
- **`Netlify-Cache-Tag`** headers to purge cache by tags

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/6ccize5rzabbt2utow2w.gif)

## Demo
<!-- Share a link to your Netlify app and include some screenshots here. -->

A 10 min video explaining the app and code base.
{% embed https://www.youtube.com/watch?v=Bodr8fMbzQs %}

**Link to app**: https://market-lens-app.netlify.app/
### Cache Status Screen shots
1. Initial load of a stock page will have Cache-Status as `"Netlify Edge"; fwd=miss`
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/kloce37zaub51dbyiphe.PNG)
2. Further load of the same page will have Cache-Status as `"Netlify Edge"; hit`
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/iga5l6rrq8ua2qrup3q9.PNG)

## Platform Primitives
Tell us how you leveraged Netlify Cache Control.

To begin with I firstly configured cache as `manual` in the `netlify.toml` file for the paths I want to cache.
```
[[edge_functions]]
  function='news'
  path= '/api/news/:symbol'
  cache='manual'

[[edge_functions]]
  function='financial'
  path= '/api/financial'
  cache='manual'
```
In the above file you can see something called `symbol`, this stands for **stock ticker symbol** _(Example AAPL is the symbol for Apple INC )_. This symbol plays a key role for the caching mechanism in this project. Let see how. 

For the sake of learning I have used two approaches for querying data
1: Ticker symbol in Path Params. 
```
api/news/AAPL -> will fetch all news article related to Apple stock
```

2: Ticker symbol in Query Params. 
```
api/financial?symbol=AAPL -> will fetch financial data for Apple stock
```
### 1. Caching Response when dynamic value in Path params
For fetching the news data the ticker symbol is passed in the URL's path param. The response header looks like below.
```
const headers = {
  'Content-Type': 'application/json',
  'Cache-Control': 'public, max-age=0, must-revalidate',
  'Netlify-CDN-Cache-Control':'public, max-age=120, stale-while-revalidate=300',
  'Cache-Tag': `${symbol}, ${symbol}-news`,
};
```
I have used `Netlify-CDN-Cache-Control` with **stale-while-revalidate**
The [stale-while-revalidate](https://docs.netlify.com/platform/caching/#stale-while-revalidate-directive) in the Netlify cache header allows cached content to be served even after it expires (max-age), while simultaneously refreshing the cache in the background. In this case, it allows serving stale content for up to 300 seconds while a new version is fetched and revalidated.

When the dynamic variable (the stock symbol) is used as a path param you dont have to do much as the symbol is part of the url path.
```
/api/news/AAPL -> Will have Apple news article Cached
/api/news/MSFT -> -> Will have Microsoft news Article Cached
```
The challenge is when symbol is passed in query params. lets see how its cached.

### 2. Caching with Netlify-Vary | Dynamic value in Query params
When the symbol is passed in query params then you have to add a custom header `Netlify-Vary` in-order to cache the response. The response header looks like below.
```
export default async (req: Request, context: Context) => {
  const parsedUrl = new URL(req.url);
  const queryParams = Object.fromEntries(parsedUrl.searchParams.entries());
  const symbol = queryParams.symbol
...
const headers = {
      'Content-Type': 'application/json',
      'Access-Control-Allow-Origin': '*',
      'Cache-Control': 'public, max-age=0, must-revalidate',
      'Netlify-CDN-Cache-Control':
        'public, max-age=604800 , stale-while-revalidate=2592000',
      'Netlify-Cache-Tag': `${symbol}, ${symbol}-financial`,
      'Netlify-Vary': 'query=symbol',
    };

```
This header object is similar to the one we saw above except that we have added an additional header field **Netlify-Vary** with value `query=symbol`. This makes sure to cache response taking the `symbol` query param into consideration.
```
/api/financial?symbol=MSFT -> Will cache Microsoft financial data
/api/financial?symbol=TSLA-> Will cache Tesla's financial data
```
[More on Netlify-Vary by query params](https://docs.netlify.com/platform/caching/#vary-by-query-parameter)

## On-demand Cache Purging
I have used Netlify's [purgeCache() helper](https://docs.netlify.com/platform/caching/#use-a-function-with-the-purgecache-helper-to-purge-by-site) to implement on-demand cache invalidation. I have made this function accessible under the path `/manual-purge`
Cache invalidation is one of the hardest things to do in Software development but I'm pretty impressed how Netlify has made it easy to implement with the helper function and cache tags.
### Cache Tags
You may have noticed a field in my response headers labeled `Netlify-Cache-Tag`. This field is crucial for managing cache invalidation behavior effectively.
```
// news.ts
const header ={
....
'Netlify-Cache-Tag': `${symbol}, ${symbol}-news`,
...
}
// financial.ts
const header ={
....
'Netlify-Cache-Tag': `${symbol}, ${symbol}-financial`,
...
}
```
The use of *ticker symbol* as tag names allows us to target specific company data for purging without affecting other company data.

The below tables might help you understand how responses are tagged

| URL                        | Cache tags           |
|----------------------------|----------------------|
| /api/news/MSFT             | MSFT, MSFT-news      |
| /api/financial?symbol=MSFT | MSFT, MSFT-financial |
| /api/news/TSLA | TSLA, TSLA-news|

This granular approach ensures that updates or modifications related to particular stocks or financial information are accurately purged without affecting other data.

Example
1. `/manual-purge?tag=MSFT`
Will purge all cached response related to Microsoft i.e News, Financial and Price data
2. `/manual-purge?tag=MSFT-news`
This will purge only the news data related to Microsoft while other cached data related to Microsoft or any other company is not disturbed.
What's surprising is that the code to purge by tags is relatively simple.
```
  const url = new URL(req.url);
  const cacheTag = url.searchParams.get('tag');
  await purgeCache({
    tags: [cacheTag],
  });
```
Below image shows how cached response related to Apple stock are purged
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/00icex1nccnqvtne2ieo.PNG)

Below image shows how only cached response related to Microsoft news are purged

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/qckuiwg7yyf7uglxcsqf.PNG)

<!-- Did you implement additional platform primitives like the Netlify Image CDN or Blobs? Tell us about that too! You may qualify for more than one prompt. -->

<!-- Team Submissions: Please pick one member to publish the submission and credit teammates by listing their DEV usernames directly in the body of the post. -->

### Personal Hurdles
I have been searching API providers for stock market data and 
settled with Polygon io as the free plan allows 5 calls per minute.

To be honest, this limitation guided my cache strategy to prioritize minimizing external API calls. By serving cached data for previously accessed stocks, the app successfully avoids redundant calls to the same company.

# Conslusion
If you have reached this part of the article it would be great if you to leave a like ❤️ for the post as it would motivate me to experiment more stuffs like this.
As someone implementing edge functions and caching for the first time I found Netlify's documentation comprehensive, sparing me the need to rely heavily on YouTube tutorials. The cache-status in response headers simplified debugging not to mention the real time logs in Netlify console. The ability to group and purge cache using Netlify-cache-tags struck me as a particularly cool feature. Looking forward to use them in real life application.

Thanks to Dev.to team and Netlify for the Hackathon. I learned something new.

<!-- Don't forget to add a cover image (if you want). -->


<!-- Thanks for participating! -->