Repository: pytorch
Issue: Better testing of the autograd engine
Labels: module: autograd,triaged,better-engineering

This is to  add more tests to the autograd engine to better test edge cases to make sure we don't introduce regression in future changes.

### Determinism

Idea: The engine should be deterministic if we assume that each Node has a constant runtime. This is not guaranteed determinism but a best effort approach where most user should never see the engine being non-deterministic.


* Reproduce the non-deterministic case
    * 3 different branches, A, B and C.
    * A and B are on CPU, C is on GPU.
    * A always finish first
    * Compare the result if B finish after C and if C finish before B.
    * there should be a 1-bit difference

* Reproduce the determinism for CPU only
    * 3 CPU branches A, B and C.
    * Test when B is faster than C and C is faster than B
    * Same test with one OP per branch and >1 OP per branch (with first sync done on the first one)
    * All should aways give the same bit-wise result

### Re-entrant

* Test simple re-entrant (@wanchaol):
    * from CPU task
        * Re-entrant work finishes on CPU
        * Re-entrant work finishes on GPU
    * from GPU task
        * Re-entrant work finishes on CPU
        * Re-entrant work finishes on GPU
    * AccumulateGrad Variable Hook (see code sample below)
    * Node pre-hook (by registering a hook on a non-leaf in python)
* Test stack overflow escape mechanism (@wanchaol)
    * Once (we should already have one test for this)
    * Multiple times in a row
        * Check that re-using the worker in the pool works fine
* Mixed with errors (@pritamdamania87)
    * Reentrant on same device
        * ~~Make parent error before child finishes (is that possible?)~~
        * ~~Make child error before parent finishes~~
    * Reentrant on different devices (test both the base being CPU and base being a GPU)
        * Make parent error before child finishes
        * Make child error before parent finishes
* Serialization (@gqchen) 
    * Make sure reentrant that contains work on a different device does not block parent work on this device
* Child backward  shares a subset of the parent graph (@albanD)
    * Make sure it runs fine
    * Make sure we don’t enter the same Node at the same time (can use time.sleep(0) to yield the GIL in our custom python Function to check that)

### Code sample:
- Re-entrant in hook
```python
import torch

param = torch.rand(10, requires_grad=True)

# This automatically adds my super fancy gradient penalty to the gradients
handle = None
def add_gradient_penalty_to_grad(grad):
    handle.remove()
    old_param_grad = param.grad
    param.grad = None
    # Add some sort of gradient penalty by directly updating the gradients
    with torch.enable_grad():
        g = grad.detach().requires_grad_()
        out = ((g*2) + param).sum()
        out.backward()
    res = g.grad + param.grad
    param.grad = old_param_grad
    return res

handle = param.register_hook(add_gradient_penalty_to_grad)

# Forward pass
tmp = (param * param)
loss = tmp.sum()
# Compute the gradients
loss.backward()
# We can now do the optimizer step
print("done")
```
cc @ezyang @SsnL @albanD @zou3519 @gqchen