sharjeelyunus/github-issues-dataset huggingface dataset:
Repository: transformers
Issue: Collection of Tokenizer issues
Labels: Discussion,WIP,bug

### System Info

```shell
Transformers + Tokenizers
```


### Who can help?

This Issue is a summary of multiple problems that we are currently encountering with Tokenizers. To solve them we'll need a more profound discussion of:
- To what extend fast and slow tokenizers should be aligned
- Whether all slow tokenizers should be kept
- How to treat special tokens
- Whether all internal methods of tokenizer should be exposed

Relevant issues/PRs:
https://github.com/huggingface/transformers/issues/15420
https://github.com/huggingface/transformers/issues/16336
https://github.com/huggingface/transformers/issues/16334
https://github.com/huggingface/transformers/issues/16337
https://github.com/huggingface/transformers/issues/15138
https://github.com/huggingface/transformers/issues/16339
https://github.com/huggingface/transformers/pull/15775

To community:
At the moment we sadly don't find the time to dive deeper here, but we're trying hard to allocate time to discuss the strategy here soon.



### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

See issues above

### Expected behavior

```shell
Don't know yet
```
Repository: go
Issue: cmd/compile: SIGILL when cross-compiling from netbsd/arm64 to linux/mips
Labels: OS-NetBSD,NeedsInvestigation,arch-arm64,compiler/runtime

```
            # internal/abi
            SIGILL: illegal instruction
            PC=0x273d40 m=4 sigcode=4
            instruction bytes: 0xa8 0x74 0x41 0x79 0x8 0x1 0x7f 0xb2 0xa8 0x74 0x1 0x79 0xd2 0xf9 0xff 0x17
            
            goroutine 9 [running]:
            cmd/internal/obj/mips.preprocess(0x4000176400, 0x4000443080, 0x40000a81a0)
            	/var/gobuilder/buildlet/go/src/cmd/internal/obj/mips/obj0.go:247 +0x1bd0 fp=0x4000637d50 sp=0x4000637bd0 pc=0x273d40
            cmd/internal/obj.Flushplist(0x4000176400, 0x4000637f28, 0x40000a81a0, {0xffffffffe6e7, 0xc})
            	/var/gobuilder/buildlet/go/src/cmd/internal/obj/plist.go:150 +0x630 fp=0x4000637e80 sp=0x4000637d50 pc=0x135610
            cmd/compile/internal/objw.(*Progs).Flush(...)
            	/var/gobuilder/buildlet/go/src/cmd/compile/internal/objw/prog.go:124
            cmd/compile/internal/ssagen.Compile(0x40003fdb80, 0x778704?)
            	/var/gobuilder/buildlet/go/src/cmd/compile/internal/ssagen/pgen.go:208 +0x574 fp=0x4000637f50 sp=0x4000637e80 pc=0x5d48b4
            cmd/compile/internal/gc.compileFunctions.func4.1(0x0?)
            	/var/gobuilder/buildlet/go/src/cmd/compile/internal/gc/compile.go:153 +0x3c fp=0x4000637f90 sp=0x4000637f50 pc=0x7785cc
            cmd/compile/internal/gc.compileFunctions.func3.1()
            	/var/gobuilder/buildlet/go/src/cmd/compile/internal/gc/compile.go:140 +0x48 fp=0x4000637fd0 sp=0x4000637f90 pc=0x778718
            runtime.goexit()
            	/var/gobuilder/buildlet/go/src/runtime/asm_arm64.s:1155 +0x4 fp=0x4000637fd0 sp=0x4000637fd0 pc=0x74814
            created by cmd/compile/internal/gc.compileFunctions.func3
            	/var/gobuilder/buildlet/go/src/cmd/compile/internal/gc/compile.go:138 +0x7c
```

`greplogs --dashboard -md -l -e \\A.\*-arm\(.\*\\n\)\*\\s\*SIGILL --since=2022-01-01`

[2022-04-28T18:02:40-c15d0a9/netbsd-arm64-bsiegert](https://build.golang.org/log/9c85452a236515691ef67b4f3608112887613eb1)

Looks like some kind of memory corruption to me, although at N=1 it isn't clear to me whether this is bug is specific to `netbsd` or `arm64`, or even some kind of weird bug in the `mips` compiler backend. 😅

(CC @golang/netbsd @golang/arm)
Repository: vscode
Issue: Local terminal can't be opened: \bin\bash not found (though VS Code running on Windows)
Labels: bug,remote,confirmed,terminal-process

<!-- Please search existing issues to avoid creating duplicates, and review our troubleshooting tips: https://code.visualstudio.com/docs/remote/troubleshooting -->
<!-- Please attach logs to help us diagnose your issue. Learn more here: https://code.visualstudio.com/docs/remote/troubleshooting#_reporting-issues -->
<!-- Also please test using the latest insiders build to make sure your issue has not already been fixed: https://code.visualstudio.com/insiders/ -->

- VSCode Version: 1.66.1
- Local OS Version: Windows 10
- Remote OS Version: Linux
- Remote Extension/Connection Type: Docker/WSL2 running on host
- Logs: error message: ```The terminal process failed to launch: Path to shell executable "\bin\bash" does not exist.```

Steps to Reproduce:

1. VS Code command: ```> Terminal: Create New Integrated Terminal (Local)```
2. The error popup appears (see above "Logs")

<!-- Check to see if the problem is general, with a specific extension, or only happens when remote -->
Does this issue occur when you try this locally?: N/A
Does this issue occur when you try this locally and all extensions are disabled?: Yes

<!-- If your issue only appears in Codespaces, please visit: https://github.com/github/feedback/discussions/categories/codespaces-feedback -->

Seems like microsoft/vscode-remote-release#730 is broken.
I checked with ```@modified terminal``` if there are any changes. Only inside the container the following settings are set:
```terminal.integrated.profiles.linux``` and ```terminal.integrated.defaultProfile.linux```.
Shouldn't it run cmd or powershell locally?
Repository: go
Issue: runtime: make underlying implementation of channels more efficient (Reference -> https://github.com/alphadose/ZenQ)
Labels: Performance,NeedsInvestigation

Current implementation of channels could use some improvements in terms of `ns/op`, `B/op` and `allocs/op`

I have made a POC thread-safe queue which has better performance in all the above 3 metrics [https://github.com/alphadose/ZenQ](https://github.com/alphadose/ZenQ)

Here are the benchmarks vs native channels [https://github.com/alphadose/ZenQ#benchmarks](https://github.com/alphadose/ZenQ#benchmarks) for a wide range of use cases

My proposal is to make the underlying implementation of native channels more efficient by changing the algorithm and data structure to the lock-free one used in ZenQ or some other suitable implementation (debatable)
Repository: angular
Issue: Enable the `no-duplicate-enum-values` TSLint rule for the FW repo
Labels: feature,area: build & ci,feature: in backlog

### Which @angular/* package(s) are relevant/related to the feature request?

core

### Description

As a followup of https://github.com/angular/angular/pull/45825#discussion_r862797290, we should consider enabling `no-duplicate-enum-values` TSLint rule to catch accidental duplicate enum values.
Repository: neovim
Issue: build: write minimal logs to stdout, full log as CI artifact
Labels: enhancement,build,test,ci

## Problem

Build logs in CI are mostly noise. But sometimes we need the full details of the log.

https://github.com/neovim/neovim/pull/17906#issuecomment-1109171265

## Solution

1. Modify `test/busted/outputHandlers/nvim.lua` so that:
    1. when running in CI, it writes the full logs (as today) to a known location
        1. make the full log available as a GHA artifact.
    1. when running in CI, it writes _minimal_ logs to stdout
        1. omits _successful_ tests
        1. prints a dot (.) every few tests (to avoid "hang" detection in CI)
1. Modify `src/nvim/testdir/runnvim.sh` in a similar way (or pipe it through the same filter as `test/busted/outputHandlers/nvim.lua`)

---

## See also

- https://github.com/neovim/neovim/pull/11009 will reduce the "server failed to start" spam in the logs
- https://github.com/neovim/neovim/pull/8519
Repository: pytorch
Issue: [torch::deploy] Remove manager_ from AquireSession
Labels: oncall: package/deploy,imported

[torch::deploy](https://github.com/pytorch/pytorch/blob/master/docs/source/deploy.rst) is a runtime for getting pytorch models to run on multiple threads in eager mode. There's a great whitepaper about torch.package and torch::deploy [here](https://arxiv.org/pdf/2104.00254.pdf).

Motivation: We want to be able to breakout interpreter manager from the interpreter. In practice what this means is that the interpreter has no reliance on the interpreter manager. With this we will be able to ship separate `deployinterpreter.so` files. This also allows people to write their own interpreter manager for high performance use cases.

Effectively what we need to do is remove the reliance of `manager_` from `Interpreter/ InterpreterSession`

Steps
1. In `deploy.cpp` in the function`InterpreterSession ReplicatedObj::acquireSession` remove `manager_`, have it such that if you attempt to `AquireSession` on an interpreter, the session will be on that interpreter or things will fail 
2. Make corresponding changes to the function in `deploy.h`, and tests such as the ones found in `test_deploy.cpp`, `test_deploy_gpu.cpp`, etc.
3. Test by running the following commands for OSS from the `pytorch` root directory (with your pytorch conda env)

`git submodule sync && git submodule update --init --recursive`
`DEBUG=1 USE_CUDA=OFF USE_OPENMP=OFF USE_DISTRIBUTED=OFF USE_DEPLOY=ON BUILD_CAFFE2=OFF BUILD_CAFFE2_OPS=OFF python setup.py develop`
`python ~/local/pytorch/torch/csrc/deploy/example/generate_examples.py`
`./build/bin/test_deploy`
Repository: go
Issue: spec: provide some way to refer to a generic function without instantiation
Labels: NeedsInvestigation,generics

The motivation behind type aliases was the ability to refer under a new name to a thing that exists elsewhere. Prior to generics, functions can be easily re-referred to using assignment:

```go
var Cut = strings.Cut
```

However, doing the same thing with a generic function fails with error message "cannot use generic function F without instantiation". It is possible to define a wrapper function, but that requires copy/pasting the entire function signature.

It'd be nice if the plain assignment worked.

I understand that this probably raises non-trivial spec difficulties. I just wanted to flag it as a useful thing.

I imagine this is a dupe, but I couldn't find it.

cc @ianlancetaylor @dsnet
Repository: pytorch
Issue: [torch::deploy] remove reliance on manager_ for unload
Labels: oncall: package/deploy,imported

[torch::deploy](https://github.com/pytorch/pytorch/blob/master/docs/source/deploy.rst) is a runtime for getting pytorch models to run on multiple threads in eager mode. There's a great whitepaper about torch.package and torch::deploy [here](https://arxiv.org/pdf/2104.00254.pdf).

Motivation: We want to be able to breakout interpreter manager from the interpreter. In practice what this means is that the interpreter has no reliance on the interpreter manager. With this we will be able to ship separate `deployinterpreter.so` files. This also allows people to write their own interpreter manager for high performance use cases.

Effectively what we need to do is remove the reliance of `manager_` from `Interpreter/ InterpreterSession`

Steps
1. In `deploy.cpp` in the function `ReplicatedObjImpl::unload(const Interpreter* onThisInterpreter) ` remove the `onThisInterpreter` logic.
2. Make corresponding changes to the function in `deploy.h`, and tests such as the ones found in `test_deploy.cpp`, `test_deploy_gpu.cpp`, etc.
3. Test by running the following commands for OSS from the `pytorch` root directory (with your pytorch conda env)

`git submodule sync && git submodule update --init --recursive`
`DEBUG=1 USE_CUDA=OFF USE_OPENMP=OFF USE_DISTRIBUTED=OFF USE_DEPLOY=ON BUILD_CAFFE2=OFF BUILD_CAFFE2_OPS=OFF python setup.py develop`
`python ~/local/pytorch/torch/csrc/deploy/example/generate_examples.py`
`./build/bin/test_deploy`
Repository: PowerToys
Issue: Child/pop up window snapping plus centering in zone
Labels: Idea-Enhancement,Product-FancyZones,Needs-Triage

### Description of the new feature / enhancement

Add option to center child windows/popups in the zone vs snapping/maximizing.  Some windows cannot maximize and are simply snapped to the upper left corner.

### Scenario when this would be used?

Some child/pop up windows cannot be maximized (example: Outlook Notifications window).  Further ven with child windows that can maximize sometimes we don't want them to be (just the parent).  In these cases, it would be nice to keep the popup within the zone but essentially center it instead.  


### Supporting information

_No response_
Repository: pytorch
Issue: [torch::deploy] move create_movable to interpreter_manager
Labels: oncall: package/deploy,imported

[torch::deploy](https://github.com/pytorch/pytorch/blob/master/docs/source/deploy.rst) is a runtime for getting pytorch models to run on multiple threads in eager mode. There's a great whitepaper about torch.package and torch::deploy [here](https://arxiv.org/pdf/2104.00254.pdf).

Motivation: We want to be able to breakout interpreter manager from the interpreter. In practice what this means is that the interpreter has no reliance on the interpreter manager. With this we will be able to ship separate `deployinterpreter.so` files. This also allows people to write their own interpreter manager for high performance use cases.

Effectively what we need to do is remove the reliance of `manager_` from `Interpreter/ InterpreterSession`

Steps
1. In `deploy.cpp` move the function the function`ReplicatedObj InterpreterSession::createMovable(Obj obj)` to reside in the `InterpreterManager`, such that the manager uses itself to create the `ReplicatedObj`. As for which interpreterSession to use, ideally we could just pick a free one and use that in a function with the signature `InterpreterManager::createMovable(Obj obj)`. Furthermore, we'll let the user to input the InterpreterSession they want the object on as another parameter of `createMovable` by creating another function with the signature `InterpreterManager::createMovable(Obj obj, InterpreterSession session)`. 

3. Make corresponding changes to the function in `deploy.h`, and tests such as the ones found in `test_deploy.cpp`, `test_deploy_gpu.cpp`, etc.
4. Test by running the following commands for OSS from the `pytorch` root directory (with your pytorch conda env)

`git submodule sync && git submodule update --init --recursive`
`DEBUG=1 USE_CUDA=OFF USE_OPENMP=OFF USE_DISTRIBUTED=OFF USE_DEPLOY=ON BUILD_CAFFE2=OFF BUILD_CAFFE2_OPS=OFF python setup.py develop`
`python ~/local/pytorch/torch/csrc/deploy/example/generate_examples.py`
`./build/bin/test_deploy`
Repository: pytorch
Issue: [torch::deploy] Remove `manager_` from the constructor and deconstructor of `InterpreterSession`
Labels: module: deploy,oncall: package/deploy,imported

[torch::deploy](https://github.com/pytorch/pytorch/blob/master/docs/source/deploy.rst) is a runtime for getting pytorch models to run on multiple threads in eager mode. There's a great whitepaper about torch.package and torch::deploy [here](https://arxiv.org/pdf/2104.00254.pdf).

Motivation: We want to be able to breakout interpreter manager from the interpreter. In practice what this means is that the interpreter has no reliance on the interpreter manager. With this we will be able to ship separate `deployinterpreter.so` files. This also allows people to write their own interpreter manager for high performance use cases.

Effectively what we need to do is remove the reliance of `manager_` from `Interpreter/ InterpreterSession`

Steps
1. Remove `manager_` from the constructor and deconstructor of `InterpreterSession` in `deploy.cpp`. 

3. Make corresponding changes to the function in `deploy.h`, and tests such as the ones found in `test_deploy.cpp`, `test_deploy_gpu.cpp`, etc.
4. Test by running the following commands for OSS from the `pytorch` root directory (with your pytorch conda env)

`git submodule sync && git submodule update --init --recursive`
`DEBUG=1 USE_CUDA=OFF USE_OPENMP=OFF USE_DISTRIBUTED=OFF USE_DEPLOY=ON BUILD_CAFFE2=OFF BUILD_CAFFE2_OPS=OFF python setup.py develop`
`python ~/local/pytorch/torch/csrc/deploy/example/generate_examples.py`
`./build/bin/test_deploy`

cc @wconstab
Repository: flutter
Issue: Vectorize `BoxConstraints`
Labels: engine,dependency: dart,c: proposal,P3,team-engine,triaged-engine

The following is the profile for `layout_bench.dart` in Flutter.  A not insignificant amount of time is spent in `Double.clamp`. This should be vectorized.  Furthermore in `customer:money` layout time takes up ~~450ms~~ 200ms of startup time on low end android devices, so there is a lot of opportunity here.

<img width="781" alt="Screen Shot 2022-04-29 at 4 04 31 PM" src="https://user-images.githubusercontent.com/30870216/166308244-4a9e8497-fbe1-4dbf-a5de-66fe5d4af6b0.png">

There are 2 dart issues blocking this from happening:
1) https://github.com/dart-lang/sdk/issues/31487
2) https://github.com/dart-lang/sdk/issues/40427

Here is the code, we could even get to Float32x4 possibly:
```dart
  BoxConstraints enforce(BoxConstraints constraints) {
    return BoxConstraints(
      minWidth: minWidth.clamp(constraints.minWidth, constraints.maxWidth),
      maxWidth: maxWidth.clamp(constraints.minWidth, constraints.maxWidth),
      minHeight: minHeight.clamp(constraints.minHeight, constraints.maxHeight),
      maxHeight: maxHeight.clamp(constraints.minHeight, constraints.maxHeight),
    );
  }
```

related issue: https://github.com/flutter/flutter/issues/94596
Repository: go
Issue: x/net/http2: apparent deadlock in TestTransportResponseHeaderTimeout_Body on windows-386-2008
Labels: NeedsInvestigation

```
panic: test timed out after 10m0s
…
goroutine 10660 [semacquire, 9 minutes]:
sync.runtime_Semacquire(0x13397808)
	C:/workdir/go/src/runtime/sema.go:56 +0x36
sync.(*WaitGroup).Wait(0x13397800)
	C:/workdir/go/src/sync/waitgroup.go:136 +0x87
golang.org/x/net/http2.(*clientTester).run(0x130e9110)
	C:/workdir/gopath/src/golang.org/x/net/http2/transport_test.go:860 +0x190
golang.org/x/net/http2.testTransportResponseHeaderTimeout(0x130ea0f0, 0x1)
	C:/workdir/gopath/src/golang.org/x/net/http2/transport_test.go:2216 +0x151
golang.org/x/net/http2.TestTransportResponseHeaderTimeout_Body(0x130ea0f0)
	C:/workdir/gopath/src/golang.org/x/net/http2/transport_test.go:2164 +0x26
testing.tRunner(0x130ea0f0, 0x823e98)
	C:/workdir/go/src/testing/testing.go:1449 +0x113
created by testing.(*T).Run
	C:/workdir/go/src/testing/testing.go:1496 +0x374
```

`greplogs -l -e 'panic: test timed out after (?:.*\n)*golang\.org/x/net/http2\.TestTransportResponseHeaderTimeout' --since=2021-01-01`
[2022-05-01T00:24:25-2871e0c-d09ca2c/windows-386-2008](https://build.golang.org/log/ce93c3e449a526a3ec0838a97c5ee13916cefda1)

(CC @neild @tombergan)
Repository: flutter
Issue: Support larger color encodings
Labels: framework,engine,P3,team-engine,triaged-engine

Currently the framework uses the `Color` class from dart:ui which expresses all colors as a 32 bit integer. 

Problems:
 - Flutter may not be able to express all of the colors supported by newer hardware with more than 8 bits of color data per channel. 
 - Rounding errors from ints to floats throughout color lerping code.
 - The engine currently expresses opacities as a float and so does the framework. However, since both Paint and pushOpacity only support 8bit alpha we lose some precision and rounding. Fixing this only for pushOpacity leads to different behavior based on whether a framework opacity is composited or not. See https://github.com/flutter/flutter/pull/102687 and  https://github.com/flutter/flutter/issues/102888
 
 
Updates to the color API: Because all of the existing methods take ints, I've essentially introduced a corresponding set of methods with an `F` on the end. I'm not really in love with this kind of naming, but I don't think we should consider any breaking changes to the existing Color methods.
 
 ```dart

 /// Create a color from four floating point numbers between 0.0 and 1.0.
 const Color.fromRGBOF(double r, double g, double b, double o);
 
 /// The backing color components exposed as doubles
 double get redF => ...
 
 double get greenF => ...
 
 double get blueF => ...
 
 double get opacity =>...
 
 Color withRedF(double red) => ...
 
 Color withBlueF(double blue) => ...
 
 Color withGreenF(double green) => ...
 ```
 
 All of the existing byte based methods/constructors will remain and will convert their ints to doubles. The existing `int get value` will be changed to compute a byte representation on the fly. This will also give us the opportunity to reduce rounding errors in color lerping and interpolation throughout the framework.
 
 This will require work in the engine to ensure that `Paint` and other objects which accept colors use the 4 double encoding instead of the int encoding, though this does not have to be done at the same time
 
This may require updates to APIs which accept an Int32List as colors, to express larger colors they would also need to accept a Float32List. this could be done non breaking but may lead to some awkward APIs. more investigation required.
 
 This is a breaking change for classes that implement but do not extend Color, but I have not found any.