Repository: next.js
Issue: fix pageExtensions to allow regular expressions and also "draft" pages feature
Labels: bug

### Verify canary release

- [X] I verified that the issue exists in the latest Next.js canary release

### Provide environment information


    Operating System:
      Platform: darwin
      Arch: arm64
      Version: Darwin Kernel Version 22.2.0: Fri Nov 11 02:06:26 PST 2022; root:xnu-8792.61.2~4/RELEASE_ARM64_T8112
    Binaries:
      Node: 18.12.1
      npm: 8.19.2
      Yarn: 1.22.19
      pnpm: 7.24.3
    Relevant packages:
      next: 13.1.7-canary.4
      eslint-config-next: 13.1.2
      react: 18.2.0
      react-dom: 18.2.0



### Which area(s) of Next.js are affected? (leave empty if unsure)

_No response_

### Link to the code that reproduces this issue

https://github.com/y-nk/nextjs-fix-45646

### To Reproduce

1. clone the project
2. `npm install`
3. `npm run build`

→ throws an error

4. `npm run patch`
5. `npm run build`

→ it works!

### Describe the Bug

**The regular expression which encapsulate `pageExtensions` used during the build does not allow a full range of regular expression syntax to be inserted.** (otherwise said: cannot use some regexp feature in pageExtension, even as simple as `tsx?`)

As a prerequisite, be aware of the `pageExtensions` value in next config (otherwise everything below might sound crazy):

```js
 pageExtensions: process.env.NODE_ENV !== 'development'
    ? ["(?<!(drafts).*)tsx?"] // all ts + tsx pages except when `(drafts)` is in the path
    : ["tsx?"], // all ts + tsx pages
```

#### Investigation / How to fix

I've started by adding `console.log` to debug the RegExp. It seems to be assembled properly but there's a bug in webpack later on (follow me please).

Places to log:

1. `regexp`: https://github.com/vercel/next.js/blob/836ac9cc7f290e95b564a61341fa95a5f4f0327e/packages/next/src/build/index.ts#L489-L498
2. `{ absolutePath, filter, partName }`: https://github.com/vercel/next.js/blob/836ac9cc7f290e95b564a61341fa95a5f4f0327e/packages/next/src/lib/recursive-readdir.ts#L40-L42
3. `regexp`: https://github.com/vercel/next.js/blob/836ac9cc7f290e95b564a61341fa95a5f4f0327e/packages/next/src/build/entries.ts#L53-L61
4. `pageKey`: https://github.com/vercel/next.js/blob/836ac9cc7f290e95b564a61341fa95a5f4f0327e/packages/next/src/build/entries.ts#L83-L85

Log output sample:

```
{
  absolutePath: '/~/app/layout.tsx',
  filter: /^page\.(?:(?<!(drafts).*).tsx?)$/,
  partName: 'layout.tsx'
}
```

From the log above, we can see that this line here may cause a problem against lookbehind in regular expression.

https://github.com/vercel/next.js/blob/836ac9cc7f290e95b564a61341fa95a5f4f0327e/packages/next/src/lib/recursive-readdir.ts#L40-L42

If we test only the file name, we can't really exclude a part of the path easily. We can change to:

```diff
-if (!filter.test(path.name)) {
+if (!filter.test(absolutePath)) {
  return;
}
```

...and the regular expression becomes aware of the full path of the file, which can be handy but the regular expression also causes problem:

https://github.com/vercel/next.js/blob/836ac9cc7f290e95b564a61341fa95a5f4f0327e/packages/next/src/build/index.ts#L489-L498

The problematic part is `/^page\.` since it enforces that the tested string starts with `page`,  a full path wouldn't pass the test. Hopefully, only removing `^` solves this issue.

```diff
if (appDir) {
  appPaths = await nextBuildSpan
    .traceChild('collect-app-paths')
    .traceAsyncFn(() =>
      recursiveReadDir(
        appDir,
-        new RegExp(`^page\\.(?:${config.pageExtensions.join('|')})$`)
+        new RegExp(`page\\.(?:${config.pageExtensions.join('|')})$`)
      )
    )
}
```

The `\\.` is not ideal, but we can always modify the `pageExtensions` to be `(?<!(drafts).*)tsx?` which omits the dot in the extension. It's a bit less natural but kinda ok.

We can further add a console.log here to help debugging:

https://github.com/vercel/next.js/blob/836ac9cc7f290e95b564a61341fa95a5f4f0327e/packages/next/src/build/entries.ts#L320-L322

Running `yarn build` shows this log, a sign that our routes are properly detected/excluded.

```
{
  appPathsPerRoute: {
    '/': [ '/page' ],
    '/article-1': [ '/(published)/article-1/page' ],
  }
}
```

...but now webpack complains:

```
TypeError: Cannot read properties of null (reading 'request')
    at /~/node_modules/next/dist/build/webpack/plugins/next-trace-entrypoints-plugin.js:220:55
```

https://github.com/vercel/next.js/blob/836ac9cc7f290e95b564a61341fa95a5f4f0327e/packages/next/src/build/webpack/plugins/next-trace-entrypoints-plugin.ts#L322

Looking at the line, it seems that `entryMod` is `null` and this is unexpected. We could eventually prevent this by adding `entryMod?.request` which is done a few lines below:

https://github.com/vercel/next.js/blob/836ac9cc7f290e95b564a61341fa95a5f4f0327e/packages/next/src/build/webpack/plugins/next-trace-entrypoints-plugin.ts#L340

...but more importantly, logging `{ compilation }` shows this:

```
ModuleNotFoundError: Module not found: Error: Can't resolve '(drafts).*)tsx%3F' in '/~/'`
```

...which seems to reveal a webpack loader string issue. Logging the `{ entrypoints }` in here:

https://github.com/vercel/next.js/blob/836ac9cc7f290e95b564a61341fa95a5f4f0327e/packages/next/src/build/webpack-build.ts#L51-L69

...shows:

```
server: {
    'app/page': {
      import: 'next-app-loader?name=app%2Fpage&pagePath=private-next-app-dir%2Fpage.tsx&appDir=%2F~%2Fapp&appPaths=%2Fpage&pageExtensions=(%3F%3C!(drafts).*)tsx%3F!',
      layer: 'sc_server'
    },
    'app/(published)/article-1/page': {
      import: 'next-app-loader?name=app%2F(published)%2Farticle-1%2Fpage&pagePath=private-next-app-dir%2F(published)%2Farticle-1%2Fpage.ts&appDir=%2F~%2Fapp&appPaths=%2F(published)%2Farticle-1%2Fpage&pageExtensions=(%3F%3C!(drafts).*)tsx%3F!',
      layer: 'sc_server'
    },
```

which confirms it. 

The import string is defined at:

https://github.com/vercel/next.js/blob/836ac9cc7f290e95b564a61341fa95a5f4f0327e/packages/next/src/build/entries.ts#L225

...and is a simple serialisation of our `pageExtensions` array. By checking if/how `pageExtensions` is used in next-app-loader, we can see this:

https://github.com/vercel/next.js/blob/836ac9cc7f290e95b564a61341fa95a5f4f0327e/packages/next/src/build/webpack/loaders/next-app-loader.ts#L193-L198

...which is obviously our problem.

Since the `import` string seems to be broken, we can eventually hotfix `pageExtensions` not to be the user defined config values, but rather an array containing the actual extension of the file (so that, it'd be a perfect match all the time):

```diff
export function getAppEntry(opts: {
  name: string
  pagePath: string
  appDir: string
  appPaths: string[] | null
  pageExtensions: string[]
  isDev?: boolean
  rootDir?: string
  tsconfigPath?: string
}) {
+ const [_, fileExtension] = opts.pagePath.match(/\.([\w]*)$/i)
+ opts.pageExtensions = [fileExtension]

  return {
    import: `next-app-loader?${stringify(opts)}!`,
    layer: WEBPACK_LAYERS.server,
  }
}
```

...but this fix creates a minor serialisation problem:

```
./node_modules/next/dist/build/webpack/loaders/next-app-loader.js?name=app%2Fpage&pagePath=private-next-app-dir%2Fpage.tsx&appDir=%2F~%2Fapp&appPaths=%2Fpage&pageExtensions=tsx!
TypeError: pageExtensions.map is not a function
    at Object.nextAppLoader (/~/node_modules/next/dist/build/webpack/loaders/next-app-loader.js:161:39)
```

it seems that, because there's only _one_ entry in `pageExtensions` now, the deserialized value of `pageExtensions` in the loader is `string` rather than `string[]`, which we can confirm by adding a `console.log` right before:

https://github.com/vercel/next.js/blob/836ac9cc7f290e95b564a61341fa95a5f4f0327e/packages/next/src/build/webpack/loaders/next-app-loader.ts#L193-L198

such as `console.log(console.log({ ...buildInfo.route, name, pageExtensions })`

...revealing that `pageExtensions` is now a string (maybe it's worth raising to webpack?):

```js
{
  page: '/page',
  absolutePagePath: '/~/app/page.tsx',
  name: 'app/page',
  pageExtensions: 'tsx' // ← not an array!
}
```

I thought first that it'd be easier to fix the `next-app-loader` to correct the type of `pageExtensions`, such as:

```js
const extensions = (Array.isArray(pageExtensions) ? pageExtensions : [pageExtensions])
  .map((extension)=>`.${extension}`);
```

...but doing so threw an Error when using non `.tsx` pages saying:

```
error - /(published)/article-1/page.ts doesn't have a root layout. To fix this error, make sure every page has a root layout.
```

...so i figured there was something about the layout being a `.tsx` and my page being a `.ts` the best way to hotfix this issue was simply:

```diff
export function getAppEntry(opts: {
  name: string
  pagePath: string
  appDir: string
  appPaths: string[] | null
  pageExtensions: string[]
  isDev?: boolean
  rootDir?: string
  tsconfigPath?: string
}) {
+ const [_, fileExtension] = opts.pagePath.match(/\.([\w]*)$/i)
+ opts.pageExtensions = ['tsx', fileExtension]

  return {
    import: `next-app-loader?${stringify(opts)}!`,
    layer: WEBPACK_LAYERS.server,
  }
}
```

### Expected Behavior

Doing all the fixes above allowed me to easily have lookbehind expressions in `pageExtensions` to create "draft" pages, only available on dev mode, but not in production mode.

Fixing the regular expression treatment also allowed me to set `pageExtensions` as `['tsx?']` instead of `['ts', 'tsx']`.

Result of the build afterwards is:

```
info  - Generating static pages (2/2)
info  - Finalizing page optimization

Route (app)                                Size     First Load JS
┌ ○ /                                      0 B                0 B
└ ○ /article-1                             0 B                0 B
```

(Don't mind the 0B, this is another bug but non blocking)

Obviously there are a few caveat in my fixing but it's a hotfix to show the steps where I've modified code. I _think_ that the correlation of `pageExtensions` and system pages of appDir (layout, error, etc) is a mistake and should be separated.

It's probably better to have it hardcoded in `getAppEntry` with something like:

```js
const NEXT_EXTENSIONS = ['js', 'jsx', 'ts', 'tsx']
const [_, fileExtension] = opts.pagePath.match(/\.([\w]*)$/i)
opts.pageExtensions = [...NEXT_EXTENSIONS, fileExtension]
```



### Which browser are you using? (if relevant)

_No response_

### How are you deploying your application? (if relevant)

_No response_