Repository: tensorflow
Issue: Out of memory in some tests due to GPU memory limit confusion
Labels: stat:awaiting tensorflower,type:bug,comp:gpu,TF 2.9,comp:lite-tosa

**System information**
- Have I written custom code (as opposed to using a stock example script provided in TensorFlow): No
- OS Platform and Distribution (e.g., Linux Ubuntu 16.04): CentOS 7
- TensorFlow installed from (source or binary): source
- TensorFlow version (use command below): 2.4.0rc4
- Python version: 3.7.4
- Bazel version (if compiling from source): 3.4.1
- GCC/Compiler version (if compiling from source): GCC 8.3.0
- CUDA/cuDNN version: 10.1
- GPU model and memory: V100

**Describe the current behavior**

I have V100 GPUs with ~32GB memory. During startup of the test (many tests show this) I see lines like 
` W tensorflow/stream_executor/stream_executor_pimpl.cc:490] Not enough memory to allocate 31614597888 on device 0 within provided limit. [used=0, limit=1073741824]`

Some tests then fail after allocating about 1GB of memory trying to allocate more. The failure message includes the 31GB and shows almost 1GB as used.
E.g. //tensorflow/python/keras/applications:applications_test or //tensorflow/python/keras/layers:convolutional_recurrent_test

**Standalone code to reproduce the issue**
Run bazel test

**Other info / logs**
```
2020-12-14 12:27:32.863559: W tensorflow/stream_executor/stream_executor_pimpl.cc:490] Not enough memory to allocate 30637607680 on device 0 within provided limit. [used=0, limit=1073741824]
2020-12-14 12:27:32.863607: W tensorflow/stream_executor/stream_executor_pimpl.cc:490] Not enough memory to allocate 30637607680 on device 0 within provided limit. [used=0, limit=1073741824]
2020-12-14 12:27:42.864373: W tensorflow/stream_executor/stream_executor_pimpl.cc:490] Not enough memory to allocate 30637607680 on device 0 within provided limit. [used=0, limit=1073741824]
2020-12-14 12:27:42.864404: W tensorflow/stream_executor/stream_executor_pimpl.cc:490] Not enough memory to allocate 30637607680 on device 0 within provided limit. [used=0, limit=1073741824]
2020-12-14 12:27:42.864421: W tensorflow/core/common_runtime/bfc_allocator.cc:433] Allocator (GPU_0_bfc) ran out of memory trying to allocate 4.00MiB (rounded to 4194304)requested by op Tanh
Current allocation summary follows.
2020-12-14 12:27:42.864434: I tensorflow/core/common_runtime/bfc_allocator.cc:972] BFCAllocator dump for GPU_0_bfc
...
2020-12-14 12:27:42.866982: I tensorflow/core/common_runtime/bfc_allocator.cc:1040] Sum Total of in-use chunks: 928.00MiB
2020-12-14 12:27:42.866989: I tensorflow/core/common_runtime/bfc_allocator.cc:1042] total_region_allocated_bytes_: 976990208 memory_limit_: 31614597888 available bytes: 30637607680 curr_region_allocation_bytes_: 63229195776
2020-12-14 12:27:42.867000: I tensorflow/core/common_runtime/bfc_allocator.cc:1048] Stats: 
Limit:                     31614597888
InUse:                       973083648
MaxInUse:                    973083648
NumAllocs:                        2238
MaxAllocSize:                 86081536
Reserved:                            0
PeakReserved:                        0
LargestFreeBlock:                    0

2020-12-14 12:27:42.867013: W tensorflow/core/common_runtime/bfc_allocator.cc:441] ****************************************************************************************************
2020-12-14 12:27:42.867035: W tensorflow/core/framework/op_kernel.cc:1763] OP_REQUIRES failed at cwise_op_gpu_base.cc:97 : Resource exhausted: OOM when allocating tensor with shape[32,32,32,32] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc

```