Repository: pytorch
Issue: Can't properly implement backward method for custom op in C++ when the op takes List of tensors as argument
Labels: module: cpp,triaged,module: custom-operators

### 🐛 Describe the bug

We have to define our custom ops in C++ because I need to run them in C++ (see https://github.com/pytorch/pytorch/issues/143773).
I was able to port most of our custom ops from Python to C++ successfully, except for one case when operation takes inputs as a list of tensors. Below is a short repro illustrating the problem. It crashes with: 
terminate called after throwing an instance of 'c10::Error'
  what():  element 0 of tensors does not require grad and does not have a grad_fn

Is there a way around it in C++ or do I have to change my custom op to take unrolled list as args ? The list have an optional entry, too. 
I was following Perplexity advice that the nested gradient list should be returned as unrolled in this case (it also wrote me the code below). 
But, it looks like we do not even get to backward being called in the first place (no printout for needs_grad).
Is there another way to do that?  @zou3519 @xmfan @yushangdi @desertfire 
Same operation defined in Python, works fine with backward returning nested list of gradients:

```
#include <torch/torch.h>
#include <vector>
#include <iostream>

class CustomFunction : public torch::autograd::Function<CustomFunction> {
public:
    // Forward pass                                                                                                                                               
    static torch::Tensor forward(torch::autograd::AutogradContext *ctx,
                                  const std::vector<torch::Tensor>& inputs) {
        // Save input tensors for backward computation                                                                                                            
        ctx->save_for_backward(inputs);

        // Example operation: sum all tensors in the list                                                                                                         
        torch::Tensor output = torch::zeros_like(inputs[0]);
        for (const auto& tensor : inputs) {
            output += tensor;
        }
        return output;
    }

    // Backward pass                                                                                                                                              
    static torch::autograd::variable_list backward(torch::autograd::AutogradContext *ctx,
                                                   torch::autograd::variable_list grad_outputs) {
        // Retrieve saved tensors                                                                                                                                 
        auto saved_inputs = ctx->get_saved_variables();
        std::cerr<<"Needs grad: "<<ctx->needs_input_grad(0)<<", "<<ctx->needs_input_grad(1)<<std::endl;
        // Compute gradients for each input tensor                                                                                                                
        torch::Tensor grad_output = grad_outputs[0];
        torch::autograd::variable_list grad_inputs;
        for (const auto& tensor : saved_inputs) {
            grad_inputs.push_back(grad_output.clone());
        }

        return grad_inputs;  // Return as a flat variable_list                                                                                                    
    }
};

int main() {
    // Create input tensors                                                                                                                                       
    auto t1 = torch::tensor({1.0}, torch::requires_grad());
    auto t2 = torch::tensor({2.0}, torch::requires_grad());
    std::vector<torch::Tensor> inputs = {t1, t2};

    // Forward pass                                                                                                                                               
    auto output = CustomFunction::apply(inputs);

    // Backward pass                                                                                                                                              
    output.backward();

    // Print gradients                                                                                                                                            
    for (size_t i = 0; i < inputs.size(); ++i) {
        std::cout << "Gradient for input " << i << ": " << inputs[i].grad() << std::endl;
    }

    return 0;
}


### Versions

Pytorch nightly.

cc @jbschlosser @zou3519 @bdhirsh @penguinwu @yf225 @chauhang