Repository: tensorflow
Issue: GPU Profiling: MemoryProfile do not contain memory events when profile remote worker.
Labels: type:bug,comp:gpu,TF 2.18

### Issue type

Bug

### Have you reproduced the bug with TensorFlow Nightly?

Yes

### Source

binary

### TensorFlow version

nightly

### Custom code

No

### OS platform and distribution

Ubuntu 22.04

### Python version

Python 3.12

### CUDA/cuDNN version

CUDA 12.4

### GPU model and memory

A100 80GB

### Current behavior?

Start a simple any collective training with Tensorflow cluster config. And then use RPC client capture_profile in Tensorboard or tf.profiler.experimental.client.trace. 
No any memory profile events or OP profiler, but only trace view.

### Standalone code to reproduce the issue

**tf_allreduce.py**
```python
import tensorflow as tf
from tensorflow.python.ops.collective_ops import all_reduce, all_reduce_v2
from tensorflow.python.eager import context
from tensorflow.core.protobuf import config_pb2
from tensorflow.python.distribute import cluster_resolver as cluster_resolver_lib

cluster_resolver = cluster_resolver_lib.TFConfigClusterResolver()
cluster = cluster_resolver.cluster_spec()
task_type = cluster_resolver.task_type
task_id = cluster_resolver.task_id

experimental_config = config_pb2.ConfigProto.Experimental(
    share_cluster_devices_in_session=False,
    share_session_state_in_clusterspec_propagation=False
)
config = config_pb2.ConfigProto(experimental=experimental_config)
config.experimental.collective_group_leader = '/job:worker/replica:0/task:0'
server = tf.distribute.Server(cluster,
                              job_name=task_type,
                              task_index=task_id,
                              protocol="grpc", # "grpc+verbs"
                              config=config)
run_options = config_pb2.RunOptions()

with tf.compat.v1.Session(target=server.target, config=config) as sess:
    tensor = tf.Variable(tf.ones([2, 2]), dtype=tf.float32)
    init = tf.compat.v1.global_variables_initializer()
    sess.run(init)
    sess.run(tf.print(["tensor:",tensor]))

    reduced_tensor = all_reduce(tensor, group_size=2, group_key=4321, instance_key=1234, merge_op='Add', final_op='Id', communication_hint='auto')
    run_options.experimental.collective_graph_key = 6
    while True:
        sess.run(tf.print(["reduced_tensor:",reduced_tensor]), options=run_options)
```

Run script to start server.
```bash
CUDA_VISIBLE_DEVICES=0 TF_CONFIG='{"cluster":{"worker":["localhost:2223","localhost:2224"]},"task":{"type":"worker","index":0}}' python tf_allreduce.py&
CUDA_VISIBLE_DEVICES=1 TF_CONFIG='{"cluster":{"worker":["localhost:2223","localhost:2224"]},"task":{"type":"worker","index":1}}' python tf_allreduce.py&
```

 use capture_profile in Tensorboard or tf.profiler.experimental.client.trace.
```python
tf.profiler.experimental.client.trace(
  'grpc://localhost:2223,grpc://localhost:2224',
   '/tmp/my_tb_dir',
   2000,
)
```

Try to convert xplane.pb to memory_profile, nothing show.
```python
from tensorflow.python.profiler.internal import _pywrap_profiler as profiler_wrapper
json = profiler_wrapper.xspace_to_tools_data(["xxx.xplane"], "memory_profile")
```

**Relevant log output**
```
{"memoryProfilePerAllocator":{},"numHosts":1,"memoryIds":[]}
```

Relative issue: #48146