Repository: tensorflow
Issue: Unable to connect to TPU through Cloud VM (metadata issue?)
Labels: type:bug,comp:tpus,TF 2.18

### Issue type

Bug

### Have you reproduced the bug with TensorFlow Nightly?

No

### Source

binary

### TensorFlow version

v2.18.0-rc2-4-g6550e4bd802 2.18.0

### Custom code

Yes

### OS platform and distribution

tpu-ubuntu2204-base

### Mobile device

_No response_

### Python version

3.11.2

### Bazel version

_No response_

### GCC/compiler version

_No response_

### CUDA/cuDNN version

_No response_

### GPU model and memory

_No response_

### Current behavior?

I am on a VM instance trying to connect to a tpu v4-32 using a test script. I installed tensorflow-tpu on both the VM (in venv) and TPU (globally) as per the instructions from the google website.

It seems like there is an issue with getting TPU metadata.

It is able to connect to the metadata server when I request manually from the VM:

```
$ curl http://169.254.169.254/computeMetadata/v1/ -H "Metadata-Flavor: Google"
instance/
oslogin/
project/
```

Any help would be appreciated!

### Standalone code to reproduce the issue

```shell
resolver = tf.distribute.cluster_resolver.TPUClusterResolver(tpu=tpu_name)
tf.config.experimental_connect_to_cluster(resolver)
try:
    tf.tpu.experimental.initialize_tpu_system(resolver)
    print("TPU initialized:", resolver.master())
except Exception as e:
    print("Failed to initialize TPU:", e)
```


### Relevant log output

```shell
$ python hello.py
2025-01-08 23:49:33.189260: I tensorflow/core/util/port.cc:153] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2025-01-08 23:49:33.221197: I tensorflow/core/tpu/tpu_api_dlsym_initializer.cc:95] Opening library: /home/ucsdwanglab/test_tpu/.venv/lib/python3.11/site-packages/tensorflow/python/platform/../../libtensorflow_cc.so.2
2025-01-08 23:49:33.221290: I tensorflow/core/tpu/tpu_api_dlsym_initializer.cc:121] Libtpu path is: /home/ucsdwanglab/test_tpu/.venv/lib/python3.11/site-packages/libtpu/libtpu.so
Failed to get TPU metadata (tpu-env) from instance metadata for variable CHIPS_PER_HOST_BOUNDS: INTERNAL: Failed to fetch URL after 30 tries (http status: 404); curl status: No error
=== Source Location Trace: === 
learning/45eac/tfrc/runtime/gcp_metadata_utils.cc:93
learning/45eac/tfrc/runtime/env_var_utils.cc:50

Failed to get TPU metadata (tpu-env) from instance metadata for variable HOST_BOUNDS: INTERNAL: Failed to fetch URL after 30 tries (http status: 404); curl status: No error
=== Source Location Trace: === 
learning/45eac/tfrc/runtime/gcp_metadata_utils.cc:93
learning/45eac/tfrc/runtime/env_var_utils.cc:50

Failed to get TPU metadata (tpu-env) from instance metadata for variable ALT: INTERNAL: Failed to fetch URL after 30 tries (http status: 404); curl status: No error
=== Source Location Trace: === 
learning/45eac/tfrc/runtime/gcp_metadata_utils.cc:93
learning/45eac/tfrc/runtime/env_var_utils.cc:50

Failed to get TPU metadata (tpu-env) from instance metadata for variable WRAP: INTERNAL: Failed to fetch URL after 30 tries (http status: 404); curl status: No error
=== Source Location Trace: === 
learning/45eac/tfrc/runtime/gcp_metadata_utils.cc:93
learning/45eac/tfrc/runtime/env_var_utils.cc:50

Failed to get TPU metadata (accelerator-type) from instance metadata for variable TPU_ACCELERATOR_TYPE: INTERNAL: Failed to fetch URL after 30 tries (http status: 404); curl status: No error
=== Source Location Trace: === 
learning/45eac/tfrc/runtime/gcp_metadata_utils.cc:93

Failed to find host bounds for accelerator type: WARNING: could not determine TPU accelerator type, please set env var `TPU_ACCELERATOR_TYPE` manually, otherwise libtpu.so may not properly initialize.
Failed to get TPU metadata (agent-worker-number) from instance metadata for variable TPU_WORKER_ID: INTERNAL: Failed to fetch URL after 30 tries (http status: 404); curl status: No error
=== Source Location Trace: === 
learning/45eac/tfrc/runtime/gcp_metadata_utils.cc:93

Failed to get TPU metadata (worker-network-endpoints) from instance metadata for variable TPU_WORKER_HOSTNAMES: INTERNAL: Failed to fetch URL after 30 tries (http status: 404); curl status: No error
=== Source Location Trace: === 
learning/45eac/tfrc/runtime/gcp_metadata_utils.cc:93

WARNING: Logging before InitGoogle() is written to STDERR
E0000 00:00:1736380405.363400    3192 common_lib.cc:511] INVALID_ARGUMENT: Error: unexpected worker hostname 'WARNING: could not determine TPU worker hostnames or IP addresses' from env var TPU_WORKER_HOSTNAMES. Expecting a valid hostname or IP address without port number. (Full TPU workers' addr string: WARNING: could not determine TPU worker hostnames or IP addresses, please set env var `TPU_WORKER_HOSTNAMES` manually, otherwise libtpu.so may not properly initialize.)
=== Source Location Trace: === 
learning/45eac/tfrc/runtime/libtpu_init_utils.cc:173
2025-01-08 23:56:48.526584: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX512F AVX512_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
E0000 00:00:1736380609.730442    3192 context_distributed_manager.cc:762] unknown service tensorflow.WorkerService
Additional GRPC error information from remote target /job:worker/replica:0/task:0 while calling /tensorflow.WorkerService/GetStatus:
:{"created":"@1736380609.730372913","description":"Error received from peer ipv4:10.130.0.3:8470","file":"external/com_github_grpc_grpc/src/core/lib/surface/call.cc","file_line":1056,"grpc_message":"unknown service tensorflow.WorkerService","grpc_status":12}
E0108 23:56:49.730822322    3192 completion_queue.cc:244]    assertion failed: queue.num_items() == 0
https://symbolize.stripped_domain/r/?trace=7f1ccaf5cebc,7f1ccaf0e04f&map= 
*** SIGABRT received by PID 3192 (TID 3192) on cpu 4 from PID 3192; stack trace: ***
PC: @     0x7f1ccaf5cebc  (unknown)  (unknown)
    @     0x7f1caa302841       1888  (unknown)
    @     0x7f1ccaf0e050   18460496  (unknown)
    @     0x7f1ccaed1c60  (unknown)  (unknown)
https://symbolize.stripped_domain/r/?trace=7f1ccaf5cebc,7f1caa302840,7f1ccaf0e04f,7f1ccaed1c5f&map= 
E0108 23:56:49.732558    3192 coredump_hook.cc:316] RAW: Remote crash data gathering hook invoked.
E0108 23:56:49.732569    3192 coredump_hook.cc:355] RAW: Skipping coredump since rlimit was 0 at process start.
E0108 23:56:49.732575    3192 client.cc:269] RAW: Coroner client retries enabled, will retry for up to 30 sec.
E0108 23:56:49.732580    3192 coredump_hook.cc:411] RAW: Sending fingerprint to remote end.
E0108 23:56:49.732595    3192 coredump_hook.cc:420] RAW: Cannot send fingerprint to Coroner: [NOT_FOUND] stat failed on crash reporting socket /var/google/services/logmanagerd/remote_coredump.socket (Is the listener running?): No such file or directory
E0108 23:56:49.732601    3192 coredump_hook.cc:472] RAW: Dumping core locally.
E0108 23:56:49.745981    3192 process_state.cc:805] RAW: Raising signal 6 with default behavior
Aborted
```