Repository: next.js
Issue: [WORKAROUND] Vercel deployment / build pipeline using filesystem cache handler, not Vercel Data Cache
Labels: bug,Pages Router

### Link to the code that reproduces this issue

https://github.com/evankirkiles/nextjs-broken-data-cache-demo

### To Reproduce

1. Clone the repo
2. Remove the `source ./get-sc-creds.sh;` part of the build script
3. Deploy to Vercel
4. See the first time displayed. Call this `Time A`.
5. Click "Revalidate" several times and see new times for each revalidation displayed, after `Time A`.
6. Change `currHash` locally in `page.tsx:5` to something different
7. Commit and push the changes, such that the build re-deploys with build cache
8. Once the deployment succeeds, see that the time displayed has regressed to `Time A`.

<div>
<img width="45%" alt="Screenshot 2023-10-03 at 11 36 34 AM" src="https://github.com/vercel/next.js/assets/30581915/b755dfab-9893-4102-aebd-c6f49dcf2326">
<img width="45%" alt="Screenshot 2023-10-03 at 11 36 54 AM" src="https://github.com/vercel/next.js/assets/30581915/fde56085-e3f2-426b-aca3-fe2dfcb3a172">

</div>

In the above pictures, see that the two times `3:35:18 PM` and `3:35:09 PM`, which came from manual revalidation of the `time` tag, are forgotten after the new deployment using the build cache. (The tabular data is saved in localStorage to persist across deployments).

### To Workaround

1. Clone the repo
2. Do NOT remove the `source ./get-sc-creds.sh;` part of the build script
3. Deploy to Vercel
4. Set the environment variables outlined in the repo [`README.md`](https://github.com/evankirkiles/nextjs-broken-data-cache-demo) and re-deploy.
5. See the first time displayed. Call this `Time A`.
6. Click "Revalidate" several times and see new times for each revalidation displayed, after `Time A`.
7. Change `currHash` locally in `page.tsx:5` to something different
8. Commit and push the changes, such that the build re-deploys with build cache
9. Once the deployment succeeds, see that the time displayed has _not_ regressed to `Time A`, and rather is the most recently-fetched time from the manual revalidation.

### Current vs. Expected behavior

## Current Behavior:
Deployments using the build cache only use data fetched _during_ the current and previous build processes, not data from the Data Cache. Essentially, manually-revalidated data (from `revalidateTag`, for example) is forgotten after a new deployment using the build cache.

## Expected Behavior:
New deployments using the build cache should use the most up-to-date data from the Vercel Data Cache, not just from previous build processes.

## Thoughts on the Issue

> 
> ~~I suspect this is because the Build Cache snapshots and uploads the `.next/cache` folder, which includes the `.next/cache/fetch-cache` folder (the local fetch cache). When re-using the build cache, then, the `.next/cache/fetch-cache` downloaded is that from the time of the last build—ignoring any requests manually revalidated during the deployment's lifetime. The Vercel Data Cache does _not_ seem to be used. This means that deployments using the build cache can only ever read data requested during the deployment build process and thus saved to the snapshotted `.next/cache/fetch-cache`—not data that has been revalidated during the lifecycle of the deployment.~~
> 
> ~~Current workarounds are:~~
> 1. ~~If using manually revalidated data, do not use the build cache. This ensures that requests are always up-to-date at build time, as data regression is far more harmful than longer build times and more requests.~~
> 2. ~~Manually revalidate all requests after a new deployment. This is a terrible solution, though, as it's a pain to manually revalidate all tags and the old data is still visible while those requests are revalidated.~~

EDIT: The problem is caused by the environment variables `SUSPENSE_CACHE_URL`, `SUSPENSE_CACHE_ENDPOINT`, and `SUSPENSE_CACHE_AUTH_TOKEN` not being set in the deployment pipeline. Without these, the fetch cache is not used, and the build pipeline always resorts to the filesystem cache in `.next/cache/fetch-cache` from the downloaded Build Cache.

## Proposed Solution

> ~~(Assuming that the above is what's going on)—don't save the `.next/cache/fetch-cache` folder in the Build Cache, instead sourcing data from Vercel's Data Cache, where manually-revalidated data is accessible. This separation between build artifacts and fetch data makes more sense to me architecturally, as well.~~

EDIT: Expose the `SUSPENSE_CACHE_URL`, `SUSPENSE_CACHE_ENDPOINT`, and `SUSPENSE_CACHE_AUTH_TOKEN` environment variables to the build pipeline in the same way that they are set for route handlers in the `ctx._requestHeaders` passed into the constructor for the `CacheHandler`.

# Workaround

In our build script, we somehow need to get `SUSPENSE_CACHE_URL`, `SUSPENSE_CACHE_ENDPOINT`, and `SUSPENSE_CACHE_AUTH_TOKEN`. These are only easily acquirable on the user-side from an existing `FetchCache` `CacheHandler`. We can get one of these from within a route handler on the global `globalThis.__incrementalCache` object. So we can populate these environment variables with three things:
1. A protected route handler that reads the above credentials and endpoint and returns them in its response
2. A bash script which calls that route handler from a previous deployment and populates the `build` process's environment variables dynamically
3. Modifying our `build` script in `package.json` to call the bash script.

This works, and once you set it up you don't have to touch it again. But it would be much better if Vercel would do this for us.

For sample code, see: https://github.com/evankirkiles/nextjs-broken-data-cache-demo.

### Verify canary release

- [X] I verified that the issue exists in the latest Next.js canary release

### Provide environment information

```bash
Operating System:
  Platform: darwin
  Arch: x64
  Version: Darwin Kernel Version 22.5.0: Mon Apr 24 20:51:50 PDT 2023; root:xnu-8796.121.2~5/RELEASE_X86_64
Binaries:
  Node: 20.3.1
  npm: 9.6.7
  Yarn: 1.22.19
  pnpm: N/A
Relevant Packages:
  next: 13.5.4-canary.11
  eslint-config-next: 13.5.4
  react: 18.2.0
  react-dom: 18.2.0
  typescript: 5.2.2
Next.js Config:
  output: N/A
```


### Which area(s) are affected? (Select all that apply)

Data fetching (gS(S)P, getInitialProps)

### Additional context

_No response_