Repository: rainx0r/metaworld-algorithms
Issue: Possible memory leak when running examples/multi_task/mtsac_mt10.py
Root cause: Resource Mishandling

### Description
I’m encountering steadily increasing memory usage when running examples/multi_task/mtsac_mt10.py, which eventually leads to the process being killed (OOM). This appears consistent with a memory leak rather than expected training-time growth.

### Expected behaviour
Memory usage should stabilize after initialization / warm-up
Long-running training should not continuously consume additional memory

### Environment
Python:  3.12.12
CUDA: 12

Please let me know if you’d like me to test a specific branch, add memory profiling output, or try potential fixes.
Thanks for the great work on this repo!

Hi, I have seen increasing memory use right around when the evaluation step kicks in because it initialises new environments, but in the training runs we did for the Meta-World+ paper and other publications I didn't notice any memory leaks. Usually requesting just enough memory on SLURM after the evaluation step is executed once was enough for it to not go OOM again.

First off, just to make sure, you are using uv and fetching the the package versions defined in the `uv.lock` file right? Because maybe the issue could be that untested newer versions of the dependencies could have some issue.

Otherwise, the memory profiling output could be useful, or perhaps could you go into more detail on what's your setup for monitoring this memory leak? I want to see if I can reproduce the behaviour locally using this same tooling so I can look further into it.

Thinking about potential root causes and just putting them in this comment for documentation:

Off the top of my head: MTSAC uses a pre-allocated timestep buffer, so that should use the same memory throughout training, and I sort of doubt that JAX recompilations are triggering (we would've probably noticed the slowdown, and even if there is recompilation going on I'm not sure it'd lead to a memory leak), so more or less all the RL algorithm-specific code in this repo is unlikely to be causing it I think. The only place I can think of where increasing memory usage could be coming from is perhaps if the evaluation environments are reinstantiated and not properly garbage collected?

Otherwise it has to be on the Metaworld (or Gymnasium, or MuJoCo) side

Thanks for that incredibly quick reply.

> Usually requesting just enough memory on SLURM after the evaluation step is executed once was enough for it to not go OOM again.

I tried with 30gb and got to 9M steps, and tried with 60gb and got to 19M steps; which seems to be almost linear and larger amounts of RAM are not available to me.

> First off, just to make sure, you are using uv and fetching the the package versions defined in the uv.lock file right? Because maybe the issue could be that untested newer versions of the dependencies could have some issue.

yes it is a fresh install, just fetching the package versions defined in the uv.lock file

> Otherwise, the memory profiling output could be useful, or perhaps could you go into more detail on what's your setup for monitoring this memory leak? I want to see if I can reproduce the behaviour locally using this same tooling so I can look further into it.

I'll think about ways to provide you with more information. Right now I am just looking at the "System Memory Utilization" graphs provided in wandb and judging from the OOM error.


> I tried with 30gb and got to 9M steps, and tried with 60gb and got to 19M steps; which seems to be almost linear and larger amounts of RAM are not available to me.

Right, so even for MT50 I didn't use more than 32GB of RAM, so that confirms that you're not reporting known behaviour.

> I'll think about ways to provide you with more information. Right now I am just looking at the "System Memory Utilization" graphs provided in wandb and judging from the OOM error.

Can you share what your hardware topology looks like? (What CPU/GPU are you using?) Might provide clues for what's going on in case I can't reproduce the memory leak on my hardware. Also could you run a `uv pip freeze > venv.txt` and share that text file? That would give me a clear picture of the setup you're working with.

I'll try and reproduce the issue on my local machine using a fresh install of the project in the meantime.