Repository: pytorch
Issue: Inconsistent results between libtorch and pytorch for model with shared weights
Labels: oncall: jit

### 🐛 Describe the bug

I created a simple model with shared weights between two linear layers, then script it and run in libtorch and pytorch. The results are different.

Pytorch code (creates model and runs 5 iterations; NOTE: it has to be run before libtorch):
```python
from torch import nn
import torch


torch.manual_seed(0)


class Model(nn.Module):
    def __init__(self):
        super().__init__()
        self.n = nn.Linear(10, 10)
        self.m = nn.Linear(10, 10)

        self.n.bias = self.m.bias
        self.n.weight = self.m.weight
    
    def forward(self, x):
        x = self.n(x)
        x = self.m(x)
        return x.mean()


model = Model()
scripted_model = torch.jit.script(model)
scripted_model.save("/home/vadim/work/tmp_python/tmp30/tmp.pt")

model = torch.jit.load("/home/vadim/work/tmp_python/tmp30/tmp.pt")
optimizer = torch.optim.SGD(model.parameters(), 0.001)
x = torch.ones(size=(10, 10))

for _ in range(5):
    optimizer.zero_grad()
    loss = model(x)
    loss.backward()
    optimizer.step()
    print(loss)
```

Libtorch:
```cpp
#include <torch/torch.h>
#include <torch/script.h>

#include <iostream>
#include <fstream>

int main() {
    torch::jit::script::Module model = torch::jit::load("tmp.pt");
    
    std::vector<at::Tensor> parameters;
    for (const auto& param : model.parameters()) {
	    parameters.push_back(param);
    }
    
    torch::optim::SGD optimizer(parameters, 0.001);
    
    torch::Tensor x = torch::ones({10, 10});
    
    for (int i = 0; i < 5; ++i)
    {
	    optimizer.zero_grad();
	    torch::Tensor loss = model.forward({ x }).toTensor();
	    loss.backward();
	    optimizer.step();
	    std::cout << "Loss: " << loss << std::endl;
    }
    
    return 0;
}
```

Pytorch result:
```shell
tensor(0.0285, grad_fn=<MeanBackward0>)
tensor(0.0278, grad_fn=<DifferentiableGraphBackward>)
tensor(0.0271, grad_fn=<DifferentiableGraphBackward>)
tensor(0.0265, grad_fn=<DifferentiableGraphBackward>)
tensor(0.0258, grad_fn=<DifferentiableGraphBackward>)
```

Libtorch result:
```shell
Loss: 0.0284842
[ CPUFloatType{} ]
Loss: 0.0271356
[ CPUFloatType{} ]
Loss: 0.0257901
[ CPUFloatType{} ]
Loss: 0.0244476
[ CPUFloatType{} ]
Loss: 0.0231082
[ CPUFloatType{} ]
```

Only the first pair of results match.

After some additional experiments:
* For model without shared weights all results match for scripted model in libtorch, scripted model in pytorch and non-scripted (original) model in pytorch.
* For model with shared weights only results for scripted model in pytorch and non-scripted (original) model in pytorch match. Moreover, results for scripted model with shared params in libtorch match results for scripted model without shared params in libtorch.

Conclusion:
In libtorch scripted model with shared parameters, that was exported from pytorch, loses the property of shared parameters (loss propagates as if no parameters are shared).

### Versions

C++:
* `libtorch 2.4.0`

Python:
* `torch==2.0.1+cu118`
* `torchvision==0.15.2+cu118`

cc @EikanWang @jgong5 @wenzhe-nrv @sanchitintel