Repository: kubernetes
Issue: In-cluster configuration fails to be properly evaluated on Windows when the current working directory is not on the system drive
Labels: kind/bug,sig/windows,help wanted,needs-triage

#### What happened:

Running `kubectl` from a container running in a Windows-based pod fails if the current working directory is on a different drive than the system drive (`C:\`):

```
PS T:\> kubectl --v=999 version
I0819 00:42:45.890463   20432 round_trippers.go:435] curl -v -XGET  -H "Accept: application/json, */*" -H "User-Agent: kubectl.exe/v1.22.0 (windows/amd64) kubernet
es/c2b5237" 'http://localhost:8080/version?timeout=32s'
I0819 00:42:47.252500   20432 round_trippers.go:454] GET http://localhost:8080/version?timeout=32s  in 1310 milliseconds
I0819 00:42:47.252500   20432 round_trippers.go:460] Response Headers:
Client Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.0", GitCommit:"c2b5237ccd9c0f1d600d3072634ca66cefdf272f", GitTreeState:"clean", BuildDate:"20
21-08-04T18:03:20Z", GoVersion:"go1.16.6", Compiler:"gc", Platform:"windows/amd64"}
I0819 00:42:47.253112   20432 helpers.go:235] Connection error: Get http://localhost:8080/version?timeout=32s: dial tcp [::1]:8080: connectex: No connection could
be made because the target machine actively refused it.
F0819 00:42:47.253112   20432 helpers.go:116] Unable to connect to the server: dial tcp [::1]:8080: connectex: No connection could be made because the target machi
ne actively refused it.
goroutine 1 [running]:
k8s.io/kubernetes/vendor/k8s.io/klog/v2.stacks(0xc000006001, 0xc00015cdc0, 0xbb, 0x125)
        /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/klog/v2/klog.go:1026 +0xbf
k8s.io/kubernetes/vendor/k8s.io/klog/v2.(*loggingT).output(0x2f9ff60, 0xc000000003, 0x0, 0x0, 0xc0009b4fc0, 0x2, 0x26abb69, 0xa, 0x74, 0x24ef00)
k8s.io/kubernetes/vendor/k8s.io/klog/v2.(*loggingT).printDepth(0x2f9ff60, 0xc000000003, 0x0, 0x0, 0x0, 0x0, 0x2, 0xc0005ee130, 0x1, 0x1)
        /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/klog/v2/klog.go:735 +0x190
k8s.io/kubernetes/vendor/k8s.io/klog/v2.FatalDepth(...)
        /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/klog/v2/klog.go:1500
k8s.io/kubernetes/vendor/k8s.io/kubectl/pkg/cmd/util.fatal(0xc00055e090, 0x8c, 0x1)
        /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/kubectl/pkg/cmd/util/helpers.go:94 +0x296
k8s.io/kubernetes/vendor/k8s.io/kubectl/pkg/cmd/util.checkErr(0x211b000, 0xc000556030, 0x1f93930)
        /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/kubectl/pkg/cmd/util/helpers.go:189 +0x948
k8s.io/kubernetes/vendor/k8s.io/kubectl/pkg/cmd/util.CheckErr(...)
        /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/kubectl/pkg/cmd/util/helpers.go:116
k8s.io/kubernetes/vendor/k8s.io/kubectl/pkg/cmd/version.NewCmdVersion.func1(0xc000589680, 0xc000451a20, 0x0, 0x1)
        /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/kubectl/pkg/cmd/version/version.go:79 +0x125
k8s.io/kubernetes/vendor/github.com/spf13/cobra.(*Command).execute(0xc000589680, 0xc000451a10, 0x1, 0x1, 0xc000589680, 0xc000451a10)
        /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:856 +0x2c2
k8s.io/kubernetes/vendor/github.com/spf13/cobra.(*Command).ExecuteC(0xc00093f180, 0xc0000dc060, 0xc0000da040, 0x3)
        /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:960 +0x375
k8s.io/kubernetes/vendor/github.com/spf13/cobra.(*Command).Execute(...)
        /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:897
main.main()
        _output/dockerized/go/src/k8s.io/kubernetes/cmd/kubectl/kubectl.go:49 +0x234

goroutine 6 [chan receive]:
k8s.io/kubernetes/vendor/k8s.io/klog/v2.(*loggingT).flushDaemon(0x2f9ff60)
        /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/klog/v2/klog.go:1169 +0x92
created by k8s.io/kubernetes/vendor/k8s.io/klog/v2.init.0
        /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/klog/v2/klog.go:420 +0xe5

goroutine 8 [select]:
k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0x1f93838, 0x2119500, 0xc000556000, 0x1, 0xc0000e0ba0)
        /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:167 +0x119
k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil(0x1f93838, 0x12a05f200, 0x0, 0x1, 0xc0000e0ba0)
        /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133 +0x9f
k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.Until(0x1f93838, 0x12a05f200, 0xc0000e0ba0)
        /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:90 +0x54
created by k8s.io/kubernetes/vendor/k8s.io/kubectl/pkg/util/logs.InitLogs
        /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/kubectl/pkg/util/logs/logs.go:51 +0x9e  
```
(Note that this call is made from a current working directory at the root of the `T:\` drive).

Running the same command from a working directory located on the system drive, however, works without issue:
```
PS C:\> kubectl --v=999 version
I0819 00:28:54.485885   46548 merged_client_builder.go:121] Using in-cluster configuration
I0819 00:28:54.597212   46548 round_trippers.go:435] curl -v -XGET  -H "Accept: application/json, */*" -H "User-Agent: kubectl.exe/v1.22.0 (windows/amd64) kubernet
es/c2b5237" -H "Authorization: Bearer <masked>" 'https://172.20.0.1:443/version?timeout=32s'
I0819 00:28:54.604207   46548 round_trippers.go:454] GET https://172.20.0.1:443/version?timeout=32s 200 OK in 5 milliseconds
I0819 00:28:54.604207   46548 round_trippers.go:460] Response Headers:
I0819 00:28:54.604207   46548 round_trippers.go:463]     Content-Type: application/json
I0819 00:28:54.604207   46548 round_trippers.go:463]     Content-Length: 277
I0819 00:28:54.604746   46548 round_trippers.go:463]     Date: Thu, 19 Aug 2021 00:28:54 GMT
I0819 00:28:54.604809   46548 round_trippers.go:463]     Cache-Control: no-cache, private
I0819 00:28:54.608939   46548 request.go:1181] Response Body: {
  "major": "1",
  "minor": "18+",
  "gitVersion": "v1.18.20-eks-8c579e",
  "gitCommit": "8c579edfc914f013ff48b2a2b2c1308fdcacc53f",
  "gitTreeState": "clean",
  "buildDate": "2021-07-31T01:34:13Z",
  "goVersion": "go1.13.15",
  "compiler": "gc",
  "platform": "linux/amd64"
}
Client Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.0", GitCommit:"c2b5237ccd9c0f1d600d3072634ca66cefdf272f", GitTreeState:"clean", BuildDate:"20
21-08-04T18:03:20Z", GoVersion:"go1.16.6", Compiler:"gc", Platform:"windows/amd64"}
Server Version: version.Info{Major:"1", Minor:"18+", GitVersion:"v1.18.20-eks-8c579e", GitCommit:"8c579edfc914f013ff48b2a2b2c1308fdcacc53f", GitTreeState:"clean",
BuildDate:"2021-07-31T01:34:13Z", GoVersion:"go1.13.15", Compiler:"gc", Platform:"linux/amd64"}
WARNING: version difference between client (1.22) and server (1.18) exceeds the supported minor version skew of +/-1
```
(Note that this command is, this time, invoked from a current working directory at the root of the `C:\` drive. The version difference warning is irrelevant to this issue -- this is a successful output.)

#### What you expected to happen:

The current working directory should have no impact on the behavior of  `kubectl`. The command should succeed regardless of the drive the current working directory is located in.

#### How to reproduce it (as minimally and precisely as possible):

* Create a Windows image with a secondary drive (e.g. using `VOLUME [ 'T:' ]` with Docker), ideally with `kubectl` installed
* Run this image in container in a pod on a Windows node
* Download/install `kubectl` (if not included with the image)
* Navigating to a directory on the secondary drive (e.g. `T:\`), run `kubectl version`
* `kubectl` fails because the in-cluster configuration is not properly evaluated

#### Anything else we need to know?:

Notice that in the working case (when invoking `kubectl` from `C:\`), this is the first line of the verbose log:
```
I0819 00:28:54.485885   46548 merged_client_builder.go:121] Using in-cluster configuration
```
This line is not present when running `kubectl` from `T:\`. 

Looking at the `client-go` project, and specifically, at `merged_client_builder.go` ([link](https://github.com/kubernetes/client-go/blob/8f2b164/tools/clientcmd/merged_client_builder.go#L120-L123)), this log is produced there:
```
	// check for in-cluster configuration and use it
	if config.icc.Possible() {
		klog.V(4).Infof("Using in-cluster configuration")
		return config.icc.ClientConfig()
	}
```

Presumably, in the failing case, that branch is not taken. Digging deeper into the implementation of `config.icc.Possible()` ([link](https://github.com/kubernetes/client-go/blob/d7ba1f2/tools/clientcmd/client_config.go#L601-L606)), it is clear why:
```
func (config *inClusterClientConfig) Possible() bool {
	fi, err := os.Stat("/var/run/secrets/kubernetes.io/serviceaccount/token")
	return os.Getenv("KUBERNETES_SERVICE_HOST") != "" &&
		os.Getenv("KUBERNETES_SERVICE_PORT") != "" &&
		err == nil && !fi.IsDir()
}
```

The path that is being evaluated -- `/var/run/secrets/kubernetes.io/serviceaccount/token` -- does not specify a drive letter (as it looks like it is assuming a Unix/Linux system).

On Windows, when a path omits a volume or drive letter, and begins with a directory separator character, it is assumed to be relative to the _current_ drive, taken from the current working directory ([reference](https://docs.microsoft.com/en-us/dotnet/standard/io/file-path-formats)).

This succeeds when running from a current working directory on the `C:\` drive, because the path `C:\var\run\secrets\kubernetes.io\serviceaccount\token` does exist. But if fails when running from another drive, as no such file exists there.

#### Environment:
- Kubernetes version (use `kubectl version`):
```
Client Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.0", GitCommit:"c2b5237ccd9c0f1d600d3072634ca66cefdf272f", GitTreeState:"clean", BuildDate:"20
21-08-04T18:03:20Z", GoVersion:"go1.16.6", Compiler:"gc", Platform:"windows/amd64"}
Server Version: version.Info{Major:"1", Minor:"18+", GitVersion:"v1.18.20-eks-8c579e", GitCommit:"8c579edfc914f013ff48b2a2b2c1308fdcacc53f", GitTreeState:"clean",
BuildDate:"2021-07-31T01:34:13Z", GoVersion:"go1.13.15", Compiler:"gc", Platform:"linux/amd64"}
```
- Cloud provider or hardware configuration: Amazon EKS
- OS (e.g: `cat /etc/os-release`):
```
PS C:\> [System.Environment]::OSVersion.Version

Major  Minor  Build  Revision
-----  -----  -----  --------
10     0      17763  0
```
- Kernel (e.g. `uname -a`): n/a
- Install tools: n/a
- Network plugin and version (if this is a network-related bug): n/a
- Others: n/a