Repository: vscode
Issue: Extension Bisect should not re-enable most extensions throughout the process
Labels: feature-request,extensions

<!-- ⚠️⚠️ Do Not Delete This! feature_request_template ⚠️⚠️ -->
<!-- Please read our Rules of Conduct: https://opensource.microsoft.com/codeofconduct/ -->
<!-- Please search existing issues to avoid creating duplicates. -->

<!-- Describe the feature you'd like. -->

This is directly related to #111669 in which the OP describes how bisect is unable to find _any_ "bad" extensions if there is more than one.

According to the docs, [bisect re-enables the extensions it considers "fine"](https://code.visualstudio.com/blogs/2021/02/16/extension-bisect). This is problematic because if you have more than one "bad" extension, it will never be able to narrow down the problem to a single extension, and it will incorrectly tell you the problem may be with VS Code.

This could be fixed by bisect not re-enabling any extensions it has removed from the problem set until one of the "bad" ones are found. This would allow you to keep finding problematic extensions by repeatedly running the bisect tool until none remain, instead of the tool only working at all when there is _only_ one "bad" extension.

# Detailed Design

Here is the example from the docs on how bisect works now:

- Bisect divides 24 extensions into two halves of 12 extensions each, and **it disables all 12 extensions of the second half.**
- In this sample, **the 8th extension is the "bad" one**, so it is the first half and not disabled. Things are still not working as we'd expect. Because there is still an issue, extension bisect repeats the process, dividing the first 12 extensions into two parts: **6 are enabled** and **6 are disabled**. **All other extensions are also re-enabled.**
...

That very last part is what is problematic. If we eliminate that step, the tool would be able to uncover any number of problematic extensions. I will give two examples of my own: one with one bad extension, and one with two. (I am not sure what bisect does when there are an odd number of extensions, so for my examples I will assume it will give the remainder to the upper half when partitioning an odd number)

- An enabled extension: …
- A disabled extension: X
- A "bad" extension: ☢️

## Example: 12 extensions with 1 bad extension

1. The bad extension is no. 3; bisect disables the upper half of all 12

| — | — | ☢️ | — | — | — | X | X | X | X | X | X |
|---|---|---|---|---|---|---|---|---|---|---|---|

2. There is still a problem So bisect disables the upper half of the lower 6

| — | — | ☢️ | X | X | X | X | X | X | X | X | X |
|---|---|---|---|---|---|---|---|---|---|---|---|

3. There is still a problem So bisect disables extensions 2 and 3

| — | X | X (☢️) | X | X | X | X | X | X | X | X | X |
|---|---|---|---|---|---|---|---|---|---|---|---|

4. Problem gone, so bisect disables the non-problematic extension (no. 1), and then re-enables the lower half of the previously disabled set (so, just the 2nd one)

| X | X | ☢️ | X | X | X | X | X | X | X | X | X |
|---|---|---|---|---|---|---|---|---|---|---|---|

5. Bisect can correctly identify no. 3 as the problematic extension

## Example: 12 extensions with 2 bad extensions

1. The bad extensions are no. 4 and 12; bisect disables the upper half of all 12

| — | — | — | ☢️ | — | — | X | X | X | X | X | X (☢️) |
|---|---|---|---|---|---|---|---|---|---|---|---|

2. There is still a problem, so bisect disables the upper half of the lower 6

| — | — | — | X (☢️) | X | X | X | X | X | X | X | X (☢️) |
|---|---|---|---|---|---|---|---|---|---|---|---|

2. There is still a problem So bisect disables the lower half of the lower 6 (the first 3). Now we are working with extensions 4-6. Bisect re-enables the lower half of this group, which is just no. 4 

| X | X | X | ☢️ | X | X | X | X | X | X | X | X (☢️) |
|---|---|---|---|---|---|---|---|---|---|---|---|

3. A problem is found with only one extension enabled, so this round of bisecting completes with a "bad" extension identified. However the user still experiences a problem so they start a new bisect. Let us assume the user has removed the first bad extension so there are now 11 extensions left, and the 11th one is the problematic one. Bisect disables the upper half of all 11 extensions (6-11)

| — | — | — | — | — | X | X | X | X | X | X (☢️) |
|---|---|---|---|---|---|---|---|---|---|---|

4. The user no longer experiences the problem, so bisect disables the un-problematic half entirely and re-enables the lower half of the remaining extensions. This repeats until the 10th extension is the only one still enabled, still no problem.

| X | X | X | X | X | — | — | — | X | X | X (☢️) |
|---|---|---|---|---|---|---|---|---|---|---|

| X | X | X | X | X | X | X | X | — | X | X (☢️) |
|---|---|---|---|---|---|---|---|---|---|---|

| X | X | X | X | X | X | X | X | X | — | X (☢️) |
|---|---|---|---|---|---|---|---|---|---|---|

5. I assume it would finally enable the last extension to confirm the user experiences a problem at all. And the user does experience it with only the 11th extension enabled, so bisect identifies it as the problem to the user.

| X | X | X | X | X | X | X | X | X | X | ☢️ |
|---|---|---|---|---|---|---|---|---|---|---|

---

Another scenario worth noting is that in the worst case, where the first and second extensions are the bad ones, the user will choose "I can reproduce" every single time. The current implementation takes this opportunity to tell the user that it could not detect a bad extension and that the problem might be with VS Code itself. In cases like this where we cannot be 100% sure that a singular extension is the problem, my new approach should probably include that same disclaimer when it identifies a potentially problematic extension.

While imperfect, this is still more useful than not identifying any extensions at all. I can think of several improvements that can be made as well; for example we could modify the algorithm to test both halves of a set of extensions for problems when a problem is encountered on one half and alert the user. This could save time by letting the user know that both halves of some set of extensions (which may be _all_ of them if it is the first try) reproduce the problem, and let them choose to continue bisecting or not.