Repository: tensorflow
Issue: Configure script automatically selects CUDA/cuDNN path instead of waiting for user input
Labels: stat:awaiting tensorflower,type:bug,type:build/install,TF 2.10

<details><summary>Click to expand!</summary> 
 
 ### Issue Type

Bug

### Have you reproduced the bug with TF nightly?

No

### Source

source

### Tensorflow Version

TF 2.10

### Custom Code

No

### OS Platform and Distribution

Fedora 37

### Mobile device

_No response_

### Python version

3.10

### Bazel version

5.3.0

### GCC/Compiler version

12.3.1

### CUDA/cuDNN version

11.8,12.1/8.0

### GPU model and memory

GTX 1660 Ti, 6 GB

### Current Behaviour?

I am having multiple CUDA versions, and I am trying to build Tensorflow from source with CUDA support.

Now the problem lays when I try to configure the build system using `./configure`. It will asks for relevant information for the build system. This includes:

1. Python path
2. Python packages path
3. Whether to support mROC
4. Whether to support CUDA
5. Whether to support TensorRT

Now, when I select CUDA support. the script seems to automatically selects my CUDA/cuDNN versions, and does not give me the possibility to select it manually, which is contradictory to what the documentation suggests at  [https://www.tensorflow.org/install/source#gpu_support](url):  _"If your system has multiple versions of CUDA or cuDNN installed, explicitly set the version instead of relying on the default"_

Now, I was able to trace the issue exactly to the `configure.py` file. 
In fact, I strongly suspects that there is a logical error on the section that parses the user input (Line 1244 on branch r2.11):
```python
  environ_save = dict(environ_cp)
  for _ in range(_DEFAULT_PROMPT_ASK_ATTEMPTS):
    if validate_cuda_config(environ_cp):
      cuda_env_names = [
          'TF_CUDA_VERSION',
          'TF_CUBLAS_VERSION',
          'TF_CUDNN_VERSION',
          'TF_TENSORRT_VERSION',
          'TF_NCCL_VERSION',
          'TF_CUDA_PATHS',
          # Items below are for backwards compatibility when not using
          # TF_CUDA_PATHS.
          'CUDA_TOOLKIT_PATH',
          'CUDNN_INSTALL_PATH',
          'NCCL_INSTALL_PATH',
          'NCCL_HDR_PATH',
          'TENSORRT_INSTALL_PATH'
      ]
      # Note: set_action_env_var above already writes to bazelrc.
      for name in cuda_env_names:
        if name in environ_cp:
          write_action_env_to_bazelrc(name, environ_cp[name])
      break

    # Restore settings changed below if CUDA config could not be validated.
    environ_cp = dict(environ_save)

    set_tf_cuda_version(environ_cp)
    set_tf_cudnn_version(environ_cp)
    if is_windows():
      set_tf_tensorrt_version(environ_cp)
    if is_linux():
      set_tf_tensorrt_version(environ_cp)
      set_tf_nccl_version(environ_cp)

    set_tf_cuda_paths(environ_cp)
```

Now, from my understanding, the script will validate the given environment, and then if that fails will ask for user input.
With that, on the first iteration of the loop, the validation will not contain the required environment variables.

I was able to solve the issue by swapping the order as follow:
```python
    environ_save = dict(environ_cp)
    for _ in range(_DEFAULT_PROMPT_ASK_ATTEMPTS):
      # Restore settings changed below if CUDA config could not be validated.
      environ_cp = dict(environ_save)

      set_tf_cuda_version(environ_cp)
      set_tf_cudnn_version(environ_cp)
      if is_windows():
        set_tf_tensorrt_version(environ_cp)
      if is_linux():
        set_tf_tensorrt_version(environ_cp)
        set_tf_nccl_version(environ_cp)

      set_tf_cuda_paths(environ_cp)
      if validate_cuda_config(environ_cp):
        cuda_env_names = [
            'TF_CUDA_VERSION',
            'TF_CUBLAS_VERSION',
            'TF_CUDNN_VERSION',
            'TF_TENSORRT_VERSION',
            'TF_NCCL_VERSION',
            'TF_CUDA_PATHS',
            # Items below are for backwards compatibility when not using
            # TF_CUDA_PATHS.
            'CUDA_TOOLKIT_PATH',
            'CUDNN_INSTALL_PATH',
            'NCCL_INSTALL_PATH',
            'NCCL_HDR_PATH',
            'TENSORRT_INSTALL_PATH'
        ]
        # Note: set_action_env_var above already writes to bazelrc.
        for name in cuda_env_names:
          if name in environ_cp:
            write_action_env_to_bazelrc(name, environ_cp[name])
        break
```


### Standalone code to reproduce the issue

```shell
Assumption: Multiple CUDA versions on /usr/local

Command:
./configure

Input Example:
1. [Default Setting]
2. [Default Setting]
3. N
4. y
5. N
```


### Relevant log output

_No response_</details>