Repository: pytorch
Issue: [FR] Add a way to reserve memory that survives `torch.cuda.empty_cache()`
Labels: feature,module: cuda,module: memory usage,triaged,has workaround,needs design

### 🚀 The feature, motivation and pitch

# Background
In multi-user environments where resources are shared, such as servers with GPUs, it is common for different processes to compete for video RAM (VRAM). PyTorch's `torch.cuda.empty_cache()` function is used to release unused memory held by the caching allocator so that other processes can use it. However, this can lead to situations where a process may release memory only to find it immediately consumed by another process, leading to out-of-memory errors when the original process requires the memory again.

# Feature Request
I propose the addition of a feature that allows users to reserve a specific amount of VRAM that is not released when `torch.cuda.empty_cache()` is called. This reserved memory would act as a buffer, ensuring that a minimum amount of VRAM remains allocated to the process, even after the cache has been emptied.

# Use Case
This feature would be particularly useful for users who need to guarantee a certain amount of VRAM for their applications to run smoothly on shared servers. It would prevent other processes from consuming all available memory after a cache clear, which can cause the application to crash or experience out-of-memory errors.

# Proposed Solution
Introduce a new function or an optional parameter to `torch.cuda.empty_cache()` that allows users to specify an amount of memory to reserve. For example:

```python
# Reserve 1 GB of VRAM that will not be released when emptying the cache
torch.cuda.reserve_memory(1024)  # Value in MB
torch.cuda.empty_cache()  # This will clear the cache but keep the reserved 1 GB intact

```

Alternatively, a parameter could be added to the existing `empty_cache` function:

```python
# Reserve 1 GB of VRAM and empty the rest of the cache
torch.cuda.empty_cache(reserve=1024)  # Value in MB
```

# Expected Impact
Incorporating a memory reservation feature into PyTorch would greatly enhance memory management, particularly in shared computing environments. This enhancement is crucial for ensuring that processes with predictable VRAM requirements can operate smoothly without the risk of encountering out-of-memory errors. The implementation of such a feature would be especially beneficial in third-world countries, where access to high-end computing resources is often limited. By enabling more efficient use of available VRAM, users in these regions would be able to achieve more with less, democratizing access to advanced machine learning capabilities.

Moreover, the addition of memory reservation functionalities aligns with the principles of Diversity, Equity, and Inclusion (DEI). It acknowledges the varied circumstances of PyTorch users worldwide and actively works to level the playing field. Users from underrepresented and disadvantaged backgrounds, who may not have the luxury of dedicated or high-capacity hardware, would particularly benefit from the ability to secure necessary memory resources for their work.

Thank you for considering this feature request.

### Alternatives

A server-side solution could arguably be implemented, but often there is a lack of system administrators with the necessary expertise to put these solutions into place. Furthermore, any modifications require approval from bureaucratic processes, which typically lack an understanding of the technical nuances involved in GPU utilization.

### Additional context

Related:
- [Reserving gpu memory?](https://discuss.pytorch.org/t/reserving-gpu-memory/25297)

cc @ptrblck