Repository: tensorflow
Issue: TF throws: "'visible_device_list' listed an invalid Device id" when using non-GPU PluggableDevices
Labels: stat:awaiting tensorflower,type:bug,comp:gpu,TF 2.13

<details><summary>Click to expand!</summary> 
 
 ### Issue Type

Bug

### Have you reproduced the bug with TF nightly?

No

### Source

source

### Tensorflow Version

2.12.0

### Custom Code

Yes

### OS Platform and Distribution

Linux CentOS

### Mobile device

_No response_

### Python version

_No response_

### Bazel version

_No response_

### GCC/Compiler version

_No response_

### CUDA/cuDNN version

11.8

### GPU model and memory

_No response_

### Current Behaviour?

When using PluggableDevice API together with GPU devices, TF crashes with `tensorflow.python.framework.errors_impl.InvalidArgumentError: 'visible_device_list' listed an invalid Device id '2' but visible device count is 2` when calling `tf.device('/GPU:0')`

**For reproducing the error, it is necessary to have a PluggableDevice Plugin loaded and to have GPUs within the same system!!!**

Here the list of devices within my system:
```python3
# tf.config.list_physical_devices()
[PhysicalDevice(name='/physical_device:CPU:0', device_type='CPU'), PhysicalDevice(name='/physical_device:VE:0', device_type='VE'), PhysicalDevice(name='/physical_device:VE:1', device_type='VE'), PhysicalDevice(name='/physical_device:GPU:0', device_type='GPU')]
```

I traced down the error to be thrown here. It gets thrown in
https://github.com/tensorflow/tensorflow/blob/e32f5b90ec16e88b23be8a5189e52ea9a420e999/tensorflow/tsl/framework/device_id_utils.cc#L46

However, it is caused by wrong values stored in the gpu_options, which get initialized here:
https://github.com/tensorflow/tensorflow/blob/0db597d0d758aba578783b5bf46c889700a45085/tensorflow/python/eager/context.py#L1206

The list of gpu_devices and ALL pluggable_devices get combined, even if they are not of the same device_type. So the list of `compatible_devices` will be:
```
[
	PhysicalDevice(name='/physical_device:GPU:0', device_type='GPU'),
	PhysicalDevice(name='/physical_device:VE:0', device_type='VE'),
	PhysicalDevice(name='/physical_device:VE:1', device_type='VE')
]
```

This causes the `visible_device_list` to be `['0', '1', '2']`, which contains invalid GPU device indices. These then get passed to `ParseVisibleDeviceList`, which throws this error.

To fix this error, it suffices to change this line:
https://github.com/tensorflow/tensorflow/blob/0db597d0d758aba578783b5bf46c889700a45085/tensorflow/python/eager/context.py#L1216

and replace it to:
```python
if dev not in gpu_devices and dev.device_type == "GPU":
```

This way, the list of `compatible_devices` will only populated with other GPUs, not with any other device types.

### Standalone code to reproduce the issue

```shell
import tensorflow as tf
print(*tf.config.list_physical_devices(), sep='\n')
tf.device('/GPU:0')
```


### Relevant log output

```shell
PhysicalDevice(name='/physical_device:CPU:0', device_type='CPU')
PhysicalDevice(name='/physical_device:VE:0', device_type='VE')
PhysicalDevice(name='/physical_device:VE:1', device_type='VE')
PhysicalDevice(name='/physical_device:GPU:0', device_type='GPU')
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/.../.local/lib/python3.8/site-packages/tensorflow/python/framework/ops.py", line 5577, in device_v2
    return device(device_name)
  File "/.../.local/lib/python3.8/site-packages/tensorflow/python/framework/ops.py", line 5526, in device
    return context.device(device_name_or_function)
  File "/.../.local/lib/python3.8/site-packages/tensorflow/python/eager/context.py", line 2348, in device
    ensure_initialized()
  File "/.../.local/lib/python3.8/site-packages/tensorflow/python/eager/context.py", line 2143, in ensure_initialized
    context().ensure_initialized()
  File "/.../.local/lib/python3.8/site-packages/tensorflow/python/eager/context.py", line 583, in ensure_initialized
    context_handle = pywrap_tfe.TFE_NewContext(opts)
tensorflow.python.framework.errors_impl.InvalidArgumentError: 'visible_device_list' listed an invalid Device id '2' but visible device count is 2
```
</details>