Repository: next.js
Issue: Page is rerendered on the server, when calling a server action after navigating back via browser button
Labels: bug,Navigation

### Link to the code that reproduces this issue

Repro with tanstack query: https://codesandbox.io/p/sandbox/nextjs-bug-13-5-4-forked-r4zfnj?file=%2Fapp%2Fcharacters.tsx%3A1%2C1

Repro without tanstack-query (for some reason, to reproduce you need to open codesandbox browser preview in a new tab and check in server console that root layout was re-rendered after browser back navigation): https://codesandbox.io/p/sandbox/nextjs-bug-13-5-4-forked-z229r9?file=%2Fapp%2Fcharacters.tsx%3A8%2C1

### To Reproduce

1. Go to root page
2. Click on any link
3. Navigate back via **browser** button
4. Scroll down to the end of the page (first call to fetchNextPage from `useInfiniteQuery`(`@tanstack-react` )  should be good, fetchNextPage calls server action)
5. Scroll further
6. Notice that page is stuck with "Fetching next page..." message

Also, please, check the attached video

https://github.com/vercel/next.js/assets/16666234/693ef577-3bc3-4075-9e51-a46b2afae2b2

### Current vs. Expected behavior

Hello, thanks for a wonderful library

I am using `useInfiniteQuery` from `@tanstack/react-query` with `next.js` and `app dir` and `server actions` to implement infinite loading pattern.

Starting from "next": "13.5.1" (on "next": "13.4.19" and below all working as expected) I experience a bug, when navigating to dynamic route via next.js Link and coming back via browser back button and then calling a server action

After browser back navigation from dynamic route and a call to `fetchNextPage` (which calls server action), fetchStatus is stuck in status fetching as well as isFetchingNextPage and isFetching are both stuck in true, also I noticed that page.tsx from the root (/) is getting rerendered on the server, which is probably the root of the problem (please, check the attached video and server console)

```js
// in useEffect

// this was working before "next": "13.5.1" (in next: 13.4.19)  now, has a described above bug
if (inView && hasNextPage && !isFetchingNextPage) {
    console.log("fetching next page!!!!!!!!!!!!!!!!!!");
    void fetchNextPage();
}

// to make it work in "next": "13.5.1" and above we need to remove `isFetchingNextPage` check
// this will work, but this is not optimal
// if (inView && hasNextPage) {
//   console.log("fetching next page!!!!!!!!!!!!!!!!!!");
//   void fetchNextPage();
// }
```

### Verify canary release

- [X] I verified that the issue exists in the latest Next.js canary release

### Provide environment information

```bash
Operating System:
  Platform: linux
  Arch: x64
  Version: #1 SMP PREEMPT_DYNAMIC Sun Aug  6 20:05:33 UTC 2023
Binaries:
  Node: 20.7.0
  npm: 10.1.0
  Yarn: 1.22.19
  pnpm: N/A
Relevant Packages:
  next: 13.5.4
  eslint-config-next: 13.5.3
  react: 18.2.0
  react-dom: 18.2.0
  typescript: 5.2.2
Next.js Config:
  output: N/A
```


### Which area(s) are affected? (Select all that apply)

App Router, Routing (next/router, next/navigation, next/link)

### Additional context

I found that if I am not using infinite loading pattern (use effect + intersection observer) and just use a button with onClick handler to append next page data to the list, seems like the issue is gone