Repository: tensorflow
Issue: Performance of nn.conv1d and keras.layers.Conv1D is low the first time any given input size is processed even if retracing is prevented! 
Labels: stat:awaiting tensorflower,comp:ops,type:performance,TF 2.8

<em>Please make sure that this is an issue related to performance of TensorFlow.
As per our
[GitHub Policy](https://github.com/tensorflow/tensorflow/blob/master/ISSUES.md),
we only address code/doc bugs, performance issues, feature requests and
build/installation issues on GitHub. tag:performance_template</em>

**System information**
- Have I written custom code (as opposed to using a stock example script provided in TensorFlow):

yes

- OS Platform and Distribution (e.g., Linux Ubuntu 16.04):

Linux Ubuntu 20.04

- TensorFlow installed from (source or binary):

pip

- TensorFlow version (use command below):

tested with TF 2.4, 2.6, 2.8

- Python version:

3.7

- CUDA/cuDNN version:

Cuda Toolkit 11.3.1
CUDNN 8.3.1 

- GPU model and memory:

Tested with GeForce GTX 1050 Ti  and GeForce GTX 1080 Ti

**Describe the current behavior**

In both cases using eager mode and using a `tf.function` with `experimental_relax_shapes=True`
running tf.nn.conv1d is slow the first time a tensor of any given size is processed and the processing of a new tensor of the same size then becomes 4 times (on GPU 1050 TI) or 10 times (on GPU 1080 Ti) faster the second or further times
.
The observed behavior is a severe problem for running inference with audio signals because audio signals generally have very different sizes and therefore in a production environment the code will run only with 10% of maximum performance (on a GPU 1080 Ti) 
for the first few 100k examples until the model has seen sufficiently many lengths to achieve peak performance.

**Describe the expected behavior**

conv1d processing time should depend on the size of the input vector and not on the number of times the same size has been seen.

**Standalone code to reproduce the issue**

colab notebook is 
[here](https://colab.research.google.com/gist/roebel/5cc15bb93566d374e87da9d2c9e885f2/performance_issue.ipynb)

**Other info / logs** Include any logs or source code that would be helpful to
diagnose the problem.

Result running on colab with GPU

```
[PhysicalDevice(name='/physical_device:CPU:0', device_type='CPU'), PhysicalDevice(name='/physical_device:GPU:0', device_type='GPU')]
=== run 1 ===
len 10000 time: 10.187 speed 9.82kHz su (10, 9991, 100)
len 10001 time: 0.603 speed 165.93kHz su (10, 9992, 100)
len 10002 time: 0.613 speed 163.08kHz su (10, 9993, 100)
len 10003 time: 0.628 speed 159.26kHz su (10, 9994, 100)
len 10004 time: 0.587 speed 170.35kHz su (10, 9995, 100)
=== run 2 ===
len 10000 time: 0.051 speed 1946.11kHz su (10, 9991, 100)
len 10001 time: 0.050 speed 2000.30kHz su (10, 9992, 100)
len 10002 time: 0.066 speed 1518.87kHz su (10, 9993, 100)
len 10003 time: 0.056 speed 1788.66kHz su (10, 9994, 100)
len 10004 time: 0.059 speed 1705.02kHz su (10, 9995, 100)
```

*Notes:*
- You notice 10 times increase the second time the inner loop is run. If you run the same notebook again the first run will be fast as well, which indicates the GPU is caching something. 
- to have the effect again your need to restart the notebook
- the behavior is not CUDA imposed because an equivalent PyTorch script runs the first and second pass of the inner loop without speed difference.