Repository: next.js
Issue: Next's fetch data store not working. Failed to set fetch cache during build but works in dev mode. Build takes hours+.
Labels: bug,Pages Router

### Link to the code that reproduces this issue

https://github.com/gauravsaini964/next-fetch-cache-failure.git

### To Reproduce

**Fetch cache failure attempt**
1. Clone repo and run build.
2. Open `.next/cache` folder.  fetch-cache will be missing.
3. Request count to origin server should be 6. Instead of 1 (or 2 afaik due to strictMode?)

**Fetch Cache success attempt**
1. Clone repo and run `pnpm dev`
2. Open localhost:3000 in browser and observe `.next/cache` folder. fetch-cache will be created.
3. Close dev mode and run `build`
4. Request count to origin server = 0.


Project Info:
1. Contains 3 routes: index, a, b
2. All routes contain 1 get fetch request to same endpoint without any header or cookie usage.
3. According to docs, only 1 origin request should be made. Rest should be hitting next's data store.

### Current vs. Expected behavior

According to docs, only 1 origin request should be made. Rest should be hitting next's data store. Which is not the case.

Fetch cache ( data store as nextjs calls it ) fails in build mode. This behaviour is widely observed across the community without any clarity. Sometimes, I get this following warning, other times there is no warning but the request aren't cached either way.

```
Failed to set fetch cache https://backend_get_request TypeError: fetch failed
    at Object.fetch (node:internal/deps/undici/undici:11372:11)
    at async globalThis.fetch (/Users/gaurav/Documents/code/nathabit/frontend/nat-store/node_modules/.pnpm/next@13.5.6_@babel+core@7.22.20_react-dom@18.2.0_react@18.2.0/node_modules/next/dist/server/lib/patch-fetch.js:199:16)
    at async invokeRequest (/Users/gaurav/Documents/code/nathabit/frontend/nat-store/node_modules/.pnpm/next@13.5.6_@babel+core@7.22.20_react-dom@18.2.0_react@18.2.0/node_modules/next/dist/server/lib/server-ipc/invoke-request.js:17:12)
    at async invokeIpcMethod (/Users/gaurav/Documents/code/nathabit/frontend/nat-store/node_modules/.pnpm/next@13.5.6_@babel+core@7.22.20_react-dom@18.2.0_react@18.2.0/node_modules/next/dist/server/lib/server-ipc/request-utils.js:45:21)
    at async /Users/gaurav/Documents/code/nathabit/frontend/nat-store/node_modules/.pnpm/next@13.5.6_@babel+core@7.22.20_react-dom@18.2.0_react@18.2.0/node_modules/next/dist/server/lib/patch-fetch.js:396:29 {
  cause: Error: read ECONNRESET
      at TCP.onStreamRead (node:internal/stream_base_commons:217:20)
      at TCP.callbackTrampoline (node:internal/async_hooks:130:17) {
    errno: -54,
    code: 'ECONNRESET',
    syscall: 'read'
  }
}
```

This leads to 1000s of extra requests to origin server because NextJS recommends to fetch data down in the component tree due to presence of data store. Structuring the app according to docs and due to this issue, build times crosses 2-3 hours.

<img width="1579" alt="Step-1 Build Run without  next" src="https://github.com/vercel/next.js/assets/16620277/6bba4cad-2026-4d90-b1bf-274242b85e99">
<img width="1576" alt="Step-2 dev mode create fetch-cache" src="https://github.com/vercel/next.js/assets/16620277/fd7f2e53-0f76-44a6-9f04-306034897899">
<img width="1575" alt="Step-3 2nd Build with  next" src="https://github.com/vercel/next.js/assets/16620277/ded999e9-d485-4dd2-a125-0421fa064122">


### Verify canary release

- [X] I verified that the issue exists in the latest Next.js canary release

### Provide environment information

```bash
Operating System:
  Platform: darwin
  Arch: arm64
  Version: Darwin Kernel Version 23.0.0: Fri Sep 15 14:41:34 PDT 2023; root:xnu-10002.1.13~1/RELEASE_ARM64_T8103
Binaries:
  Node: 18.18.2
  npm: 9.8.1
  Yarn: 1.22.19
  pnpm: 8.9.2
Relevant Packages:
  next: 13.5.7-canary.33
  eslint-config-next: 13.5.7-canary.33
  react: 18.2.0
  react-dom: 18.2.0
  typescript: 5.2.2
Next.js Config:
  output: N/A
```


### Which area(s) are affected? (Select all that apply)

App Router, Data fetching (gS(S)P, getInitialProps)

### Additional context

_No response_