Repository: pytorch
Issue: When modified the model python file, the pytorch will raise the KeyError of this file
Labels: triaged,module: data parallel

## 🐛 Bug

<!-- A clear and concise description of what the bug is. -->

## To Reproduce

When I modified the model python file when the script is running, the pytorch will raise the KeyError of this file and stop running. This bug exists in pytorch 1.4. Even I upgrade it to 1.6, it still exists yet. Here is the error information

<!-- If you have a code sample, error messages, stack traces, please provide it here as well -->
Traceback (most recent call last):
  File "main_ion.py", line 535, in <module>
  File "main_ion.py", line 311, in main
    seq_x = inputs
  File "/root/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/root/anaconda3/lib/python3.7/site-packages/torch/nn/parallel/data_parallel.py", line 155, in forward
    outputs = self.parallel_apply(replicas, inputs, kwargs)
  File "/root/anaconda3/lib/python3.7/site-packages/torch/nn/parallel/data_parallel.py", line 165, in parallel_apply
    return parallel_apply(replicas, inputs, kwargs, self.device_ids[:len(replicas)])
  File "/root/anaconda3/lib/python3.7/site-packages/torch/nn/parallel/parallel_apply.py", line 85, in parallel_apply
    output.reraise()
  File "/root/anaconda3/lib/python3.7/site-packages/torch/_utils.py", line 395, in reraise
    raise self.exc_type(msg)
KeyError: Caught KeyError in replica 0 on device 0.
Original Traceback (most recent call last):
  File "/root/anaconda3/lib/python3.7/linecache.py", line 74, in checkcache
    stat = os.stat(fullname)
FileNotFoundError: [Errno 2] No such file or directory: '/p300/projects/RT/code/IonInten/auxiliary_loss_transformer.py'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/root/anaconda3/lib/python3.7/site-packages/torch/nn/parallel/parallel_apply.py", line 60, in _worker
    output = module(*input, **kwargs)
  File "/root/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/p300/projects/RT/code/IonInten/auxiliary_loss_transformer.py", line 258, in forward
  File "/root/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/p300/projects/RT/code/IonInten/auxiliary_loss_transformer.py", line 60, in forward
  File "/root/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/p300/projects/RT/code/IonInten/auxiliary_loss_transformer.py", line 120, in forward
  File "/root/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/root/anaconda3/lib/python3.7/site-packages/torch/nn/modules/activation.py", line 927, in forward
    attn_mask=attn_mask)
  File "/root/anaconda3/lib/python3.7/site-packages/torch/nn/functional.py", line 4081, in multi_head_attention_forward
    v = v.contiguous().view(-1, bsz * num_heads, head_dim).transpose(0, 1)
  File "/root/anaconda3/lib/python3.7/traceback.py", line 197, in format_stack
    return format_list(extract_stack(f, limit=limit))
  File "/root/anaconda3/lib/python3.7/traceback.py", line 211, in extract_stack
    stack = StackSummary.extract(walk_stack(f), limit=limit)
  File "/root/anaconda3/lib/python3.7/traceback.py", line 359, in extract
    linecache.checkcache(filename)
  File "/root/anaconda3/lib/python3.7/linecache.py", line 76, in checkcache
    del cache[filename]
KeyError: '/p300/projects/RT/code/IonInten/auxiliary_loss_transformer.py'

<!-- A clear and concise description of what you expected to happen. -->

## Environment

 - PyTorch Version (e.g., 1.0):1.6
 - OS (e.g., Linux):Linux
 - How you installed PyTorch (`conda`, `pip`, source):conda
 - Build command you used (if compiling from source):
 - Python version:3.7
 - CUDA/cuDNN version:10.1
 - GPU models and configuration:
 - Any other relevant information:

## Additional context

<!-- Add any other context about the problem here. -->