Repository: vscode
Issue: [Remote-SSH] Extension Host with Copilot Fails on VS Code 1.95.3 but Works on 1.93.0
Labels: upstream,remote,nodejs,mitigated


Type: <b>Bug</b>

### Description

We are encountering an issue with the GitHub Copilot extension when using Visual Studio Code with Remote-SSH on an HPC login node. The extension works flawlessly with VS Code 1.93.0 (4849ca9bdf9666755eb463db297b69e5385090e3) but fails on VS Code 1.95.3 (f1a4fb101478ce6ec82fe9627c43efbf9e98c813). Interestingly, VS Code 1.95.3 works fine on the computing node of the same cluster, accessed via SSH forwarding from the login node.

To eliminate variables, we have tested the setup thoroughly:

- Two personal computers were used:
  - One running VS Code 1.93.0. (We didn't record the version of the extensions when we were testing, and VS Code 1.93.0 auto-updated to 1.95.3 when we tried to check the versions. Please tell us if this information is really needed. **Update:** Might be `1.245.0` / `0.20.3` from the log file `~/.vscode-server/data/logs/20241121T175642/remoteagent.log`) 
  - The other running VS Code 1.95.3, Copilot 1.245.0 and Copilot Chat 0.22.4.
- Both connect to the same HPC login node and same account using Remote-SSH.
- Before each login, the `.vscode-server` directory on the remote machine was cleaned up to ensure a fresh environment.
- On the remote machine, GitHub Copilot and GitHub Copilot Chat were the only extensions installed.


### Expected Behavior
The GitHub Copilot extension should initialize and function correctly on both VS Code versions.

### Actual Behavior

- On VS Code 1.95.3, the Copilot extension causes the Extension Host process to crash only on the login node.
- The issue does not occur on the computing node or when using VS Code 1.93.0 on the login node.
- Disabling the GitHub Copilot extensions eliminates the issue.

### Test by VS Code Bisect
Done on our login node with VS Code 1.95.3. Result:
`Extension Bisect is done and has identified github.copilot as the extension causing the problem.`

Before, we don't know that it is possible to disable GitHub Copilot alone but not GitHub Copilot Chat.
Later, we found that **when VS Code bisect disable GitHub Copilot 1.245.0 alone, GitHub Copilot Chat 0.22.4 works normally** on our login node with VS Code 1.95.3.

### Logs
Below are logs captured from the Extension Host (remote) process on both the login and computing nodes running VS Code 1.95.3. These logs highlight differences observed during initialization:

#### Login Node (Fails)
```
2024-11-21 16:34:43.523 [trace] ExtHostCommands#registerCommand github.copilot.openLogs
2024-11-21 16:34:43.523 [trace] ExtHostCommands#registerCommand github.copilot.signIn
2024-11-21 16:34:43.530 [trace] ExtensionService#_callActivateOptional GitHub.copilot-chat
2024-11-21 16:34:43.555 [trace] extHostWorkspace#findFiles2: fileSearch, extension: GitHub.copilot-chat, entryPoint: findFiles2
2024-11-21 16:34:43.556 [trace] ProxyResolver#tls.connect [{"highWaterMark":16384,"servername":"default.exp-tas.com","session":"null","localAddress":"null","ALPNProtocols":"http/1.1","port":443,"host":"default.exp-tas.com"}]
2024-11-21 16:34:43.562 [debug] ProxyResolver#resolveProxy unconfigured http://169.254.169.254/metadata/instance/compute DIRECT 
2024-11-21 16:34:43.693 [trace] ProxyResolver#tls.connect [443, "default.exp-tas.com", {"servername":"default.exp-tas.com","ALPNProtocols":"h2,http/1.1,http/1.0","signal":"[object AbortSignal]","rejectUnauthorized":true,"ca":"[281 certs]"}]
2024-11-21 16:34:43.723 [trace] ProxyResolver#tls.connect [443, "api.github.com", {"servername":"api.github.com","ALPNProtocols":"h2,http/1.1,http/1.0","signal":"[object AbortSignal]","rejectUnauthorized":true,"ca":"[281 certs]"}]
2024-11-21 16:34:43.760 [debug] ExtHostSearch /work1/user141421/.vscode-server/cli/servers/Stable-f1a4fb101478ce6ec82fe9627c43efbf9e98c813/server/node_modules/@vscode/ripgrep/bin/rg --files --hidden --case-sensitive --no-require-git -g '!**/.git' -g '!**/.svn' -g '!**/.hg' -g '!**/CVS' -g '!**/.DS_Store' -g '!**/Thumbs.db' -g '!**/node_modules' -g '!**/bower_components' -g '!**/*.code-search' --no-ignore-parent --follow --no-config --no-ignore-global
 - cwd: /home/user141421
 - Sibling clauses: {}
2024-11-21 16:34:43.964 [trace] ProxyResolver#tls.connect [443, "api.github.com", {"servername":"api.github.com","ALPNProtocols":"h2,http/1.1,http/1.0","rejectUnauthorized":true,"ca":"[281 certs]"}]
2024-11-21 16:34:43.998 [trace] ExtHostCommands#registerCommand github.copilotChat.signIn
[skip some lines]
2024-11-21 16:34:44.005 [trace] ExtHostCommands#registerCommand github.copilot.buildLocalWorkspaceIndex
2024-11-21 16:34:44.020 [debug] ExtHostSearch Search finished. Stats: {"cmdTime":266,"fileWalkTime":266,"directoriesWalked":0,"filesWalked":0,"cmdResultCount":28852}
2024-11-21 16:34:44.020 [debug] Ext host file search time: 266ms
2024-11-21 16:34:44.174 [trace] ExtHostCommands#registerCommand codereferencing.showOutputPane
2024-11-21 16:34:44.176 [trace] ProxyResolver#tls.connect [443, "copilot-telemet[39 chars]", {"servername":"copilot-telemet[39 chars]","ALPNProtocols":"h2,http/1.1,http/1.0","rejectUnauthorized":true,"ca":"[281 certs]"}]
2024-11-21 16:34:44.215 [trace] ExtHostCommands#executeCommand setContext
2024-11-21 16:34:44.215 [trace] ExtHostCommands#executeCommand _setContext
2024-11-21 16:34:44.215 [trace] ExtHostCommands#registerCommand github.copilot.generate
2024-11-21 16:34:44.215 [trace] ExtHostCommands#registerCommand github.copilot.acceptCursorPanelSolution
2024-11-21 16:34:44.215 [trace] ExtHostCommands#registerCommand github.copilot.previousPanelSolution
2024-11-21 16:34:44.215 [trace] ExtHostCommands#registerCommand github.copilot.nextPanelSolution
2024-11-21 16:34:44.216 [trace] ExtHostCommands#registerCommand _github.copilot.ghostTextPostInsert
[Process Crash]
```

#### Computing Node (Works)
```
2024-11-21 16:30:49.433 [trace] ExtHostCommands#registerCommand github.copilot.openLogs
2024-11-21 16:30:49.433 [trace] ExtHostCommands#registerCommand github.copilot.signIn
[Compare to above, no extra message here.]
2024-11-21 16:30:49.454 [trace] ExtHostCommands#registerCommand github.copilotChat.signIn
[skip the same lines as above]
2024-11-21 16:30:49.459 [trace] ExtHostCommands#registerCommand github.copilot.buildLocalWorkspaceIndex
2024-11-21 16:30:49.664 [trace] ProxyResolver#tls.connect [443, "default.exp-tas.com", {"servername":"default.exp-tas.com","ALPNProtocols":"h2,http/1.1,http/1.0","signal":"[object AbortSignal]","rejectUnauthorized":true,"ca":"[281 certs]"}]
2024-11-21 16:30:49.694 [trace] ProxyResolver#tls.connect [443, "api.github.com", {"servername":"api.github.com","ALPNProtocols":"h2,http/1.1,http/1.0","signal":"[object AbortSignal]","rejectUnauthorized":true,"ca":"[281 certs]"}]
2024-11-21 16:30:49.981 [trace] ProxyResolver#tls.connect [443, "api.github.com", {"servername":"api.github.com","ALPNProtocols":"h2,http/1.1,http/1.0","rejectUnauthorized":true,"ca":"[281 certs]"}]
2024-11-21 16:30:50.116 [trace] ExtHostCommands#registerCommand codereferencing.showOutputPane
2024-11-21 16:30:50.118 [trace] ProxyResolver#tls.connect [443, "copilot-telemet[39 chars]", {"servername":"copilot-telemet[39 chars]","ALPNProtocols":"h2,http/1.1,http/1.0","rejectUnauthorized":true,"ca":"[281 certs]"}]
2024-11-21 16:30:50.156 [trace] ExtHostCommands#executeCommand setContext
2024-11-21 16:30:50.156 [trace] ExtHostCommands#executeCommand _setContext
2024-11-21 16:30:50.156 [trace] ExtHostCommands#registerCommand github.copilot.generate
2024-11-21 16:30:50.156 [trace] ExtHostCommands#registerCommand github.copilot.acceptCursorPanelSolution
2024-11-21 16:30:50.156 [trace] ExtHostCommands#registerCommand github.copilot.previousPanelSolution
2024-11-21 16:30:50.156 [trace] ExtHostCommands#registerCommand github.copilot.nextPanelSolution
2024-11-21 16:30:50.156 [trace] ExtHostCommands#registerCommand _github.copilot.ghostTextPostInsert
2024-11-21 16:30:50.176 [debug] ProxyResolver#loadSystemCertificates count 137
2024-11-21 16:30:50.191 [debug] ProxyResolver#loadSystemCertificates count filtered 134
2024-11-21 16:30:50.191 [debug] ProxyResolver#resolveProxy unconfigured https://mobile.events.data.microsoft.com/OneCollector/1.0?cors=true&content-type=application/x-json-stream DIRECT 
2024-11-21 16:30:50.192 [trace] ProxyResolver#tls.connect [{"protocol":"https:","hostname":"mobile.events.d[32 chars]","port":443,"path":"null","method":"POST","headers":"[object Object]","agent":"[object Object]","_defaultAgent":"[object Object]","host":"mobile.events.d[32 chars]","lookupProxyAuthorization":"[Function: bound dz]","noDelay":true,"servername":"mobile.events.d[32 chars]","secureEndpoint":true,"_vscodeAdditionalCaCerts":"[134 certs]","keepAlive":true,"scheduling":"lifo","timeout":5000,"_agentKey":"mobile.events.d[57 chars]","encoding":"null","keepAliveInitialDelay":1000}]
[Process keep running without crashing]
2024-11-21 16:30:50.176 [debug] ProxyResolver#loadSystemCertificates count 137
2024-11-21 16:30:50.191 [debug] ProxyResolver#loadSystemCertificates count filtered 134
2024-11-21 16:30:50.191 [debug] ProxyResolver#resolveProxy unconfigured https://mobile.events.data.microsoft.com/OneCollector/1.0?cors=true&content-type=application/x-json-stream DIRECT 
2024-11-21 16:30:50.192 [trace] ProxyResolver#tls.connect [{"protocol":"https:","hostname":"mobile.events.d[32 chars]","port":443,"path":"null","method":"POST","headers":"[object Object]","agent":"[object Object]","_defaultAgent":"[object Object]","host":"mobile.events.d[32 chars]","lookupProxyAuthorization":"[Function: bound dz]","noDelay":true,"servername":"mobile.events.d[32 chars]","secureEndpoint":true,"_vscodeAdditionalCaCerts":"[134 certs]","keepAlive":true,"scheduling":"lifo","timeout":5000,"_agentKey":"mobile.events.d[57 chars]","encoding":"null","keepAliveInitialDelay":1000}]
```

### Additional Notes
- Network configuration remains the same across tests on both the login node and the computing node. Firewall are disable for all nodes. 
- The system administrator has confirmed that no OS-level rules (e.g., process termination) are in effect on the login node.
- All of the hardware been exactly the same for the login node and the computing nodes in our cluster.

### Key questions
- **Why it works on VS Code 1.93.0 but not on VS Code 1.95.3 ?**
- Why VS Code 1.95.3 works on our computing node but not on the login node?

VS Code version: Code 1.95.3 (f1a4fb101478ce6ec82fe9627c43efbf9e98c813, 2024-11-13T14:50:04.152Z)
OS version: Windows_NT x64 10.0.19045
Modes:
Remote OS version: Linux x64 5.15.0-78-generic

<details>
<summary>System Info</summary>

|Item|Value|
|---|---|
|CPUs|Intel(R) Core(TM) i7-7700HQ CPU @ 2.80GHz (8 x 2808)|
|GPU Status|2d_canvas: enabled<br>canvas_oop_rasterization: enabled_on<br>direct_rendering_display_compositor: disabled_off_ok<br>gpu_compositing: enabled<br>multiple_raster_threads: enabled_on<br>opengl: enabled_on<br>rasterization: enabled<br>raw_draw: disabled_off_ok<br>skia_graphite: disabled_off<br>video_decode: enabled<br>video_encode: enabled<br>vulkan: disabled_off<br>webgl: enabled<br>webgl2: enabled<br>webgpu: enabled<br>webnn: disabled_off|
|Load (avg)|undefined|
|Memory (System)|7.89GB (1.49GB free)|
|Process Argv|--log info --crash-reporter-id edf07c68-6f96-49c4-a777-c6a188cd4542|
|Screen Reader|no|
|VM|0%|

|Item|Value|
|---|---|
|Remote|SSH: Spock|
|OS|Linux x64 5.15.0-78-generic|
|CPUs|AMD Ryzen Threadripper PRO 5975WX 32-Cores (64 x 1793)|
|Memory (System)|251.70GB (244.89GB free)|
|VM|0%|
</details><details><summary>Extensions (7)</summary>

Extension|Author (truncated)|Version
---|---|---
copilot|Git|1.245.0
copilot-chat|Git|0.22.4
jupyter-keymap|ms-|1.1.2
remote-ssh|ms-|0.115.1
remote-ssh-edit|ms-|0.87.0
remote-explorer|ms-|0.4.3
vscode-speech|ms-|0.12.1

(1 theme extensions excluded)

</details><details>
<summary>A/B Experiments</summary>

```
vsliv368cf:30146710
vspor879:30202332
vspor708:30202333
vspor363:30204092
vscod805cf:30301675
binariesv615:30325510
vsaa593:30376534
py29gd2263:31024239
c4g48928:30535728
azure-dev_surveyone:30548225
962ge761:30959799
pythonnoceb:30805159
asynctok:30898717
pythonmypyd1:30879173
2e7ec940:31000449
pythontbext0:30879054
cppperfnew:31000557
dsvsc020:30976470
pythonait:31006305
dsvsc021:30996838
g316j359:31013175
dvdeprecation:31068756
dwnewjupytercf:31046870
nativerepl2:31139839
pythonrstrctxt:31112756
cf971741:31144450
iacca1:31171482
notype1cf:31157160
5fd0e150:31155592
dwcopilot:31170013
stablechunks:31184530

```

</details>

<!-- generated by issue reporter -->