Repository: vscode
Issue: Please give extensions access to native AST tokens (At least partially)
Labels: feature-request,tokenization

<!-- ⚠️⚠️ Do Not Delete This! feature_request_template ⚠️⚠️ -->
<!-- Please read our Rules of Conduct: https://opensource.microsoft.com/codeofconduct/ -->
<!-- Please search existing issues to avoid creating duplicates. -->

<!-- Describe the feature you'd like. -->

This feature request is very similar to this:
https://github.com/microsoft/vscode/issues/131062

**But now I want to reintroduce it with very updated and very carefully thought reasoning.**

Even though you might think that this feature request is too much (too unreal, too hard) to implement, I think it's worth to try because of the reasonings down below:

1] There are already too many VS Code extensions which need some access to the AST tree of editor code, and they don't have native access, so they use third party (or custom) libraries. And when syntaxes of languages get updated, they need to find new libraries (if current libraries don't keep up with the updates). And there are so many languages, so this process sooner or later will become unmanageable. VS Code probably always will have latest tokenizers/parsers natively, so let's just give extension developers access to those native AST tokens (at least partially, if not full AST).

2] One good example of such extensions is my creation <a href="https://marketplace.visualstudio.com/items?itemName=leodevbro.blockman">Blockman</a>, which has already **117K installs** and the number increases by 100-200 every day. Blockman (<a href="https://www.youtube.com/watch?v=_BedxVCzZ7o">Video intro</a>) is well loved by many people, and also many people expressed strong desire to see this extension on other IDEs too. This is a very good indicator that the visual help which Blockman provides is very desirable for so many developers.

3] Will be able to run in web (browser) environment. Many extensions like Blockman can run only in Node.js environment, because some third party libraries does not work in browser.

4] Yeah, probably this feature will be too heavy and extensions using this feature will most likely slow down entire VS Code IDE due to the fact that parsing and providing tokens is a heavy process. Well, I would say that for tokenization work - Blockman uses debouncing of 1.2 seconds by default, so it does not freeze UI and it seems acceptable (For extensions like Blockman, I think it's not necessary to have super real-time reaction on text change. 1.2 seconds debouncing seems good enough), because there are so many people who have used Blockman for many months and they still love it and they express this love with amazing comments.

These comments are probably only 1% of the amazing comments I received from many devs.
![image](https://user-images.githubusercontent.com/54504880/225867516-a509018b-7d13-41e3-a970-b8fb71d1bad1.png)


5] This VS Code API feature (providing AST of any file of any language to extensions) will open many other doors for new ideas of making the development environment much more easier and smoother and more pleasurable.


6] Get file AST: Maybe VS Code can provide smart incremental AST, so when user changes little text in a big file, the host will not re-tokenize entire file, but it will do smart incremental tokenization and then returns the AST object to the extension, so user experience will be faster/smoother (more optimized). Right now Blockman does not have any kind of smart incremental tokenization and it is still usable and still loved by many people. And think about how loved it will get if it has smart incremental tokenization.

**When I say "at least partially, if not full AST", I mean at least the locations (positions -> line, column) of brackets (curly/square/round). Of course not the brackets inside comments/strings.**

After that, it would be also good to have locations of template string tokens:
```
`
```
Then also would be good to have the locations of simple string tokens:
```
" and '
```
Then HTML/XML/JSX/TSX tags.
Then Python/Yaml indent/dedent tokens.