Repository: Cloud-CV/EvalAI
Issue: Re-running challenge submissions leads to hanging behavior
Root cause: Resource Mishandling

Hi EvalAI maintainers, thanks for all you do.

I'm a co-organizer of the "Argoverse 3D Detection Competition". I'm noticing that when I try to re-run a submission from the competition (with the "Re-run submission" button), I see "Submitted" or "Running" for the past 10+ hours, when each submission should take no more than 5 minutes to run.

Would you mind taking a look to see if the eval server is down or why jobs are not running/completing?

Thanks very much, really appreciate your efforts here.


Hi @johnwlambert ,

Thank you for reaching out.

> I'm a co-organizer of the "Argoverse 3D Detection Competition". I'm noticing that when I try to re-run a submission from the competition (with the "Re-run submission" button), I see "Submitted" or "Running" for the past 10+ hours, when each submission should take no more than 5 minutes to run.
> 
> Would you mind taking a look to see if the eval server is down or why jobs are not running/completing?


Sure, we will check this and get back to you ASAP.

Hi @johnwlambert,

I tested the evaluation worker on following two setups to figure out if the resources we allocated are insufficient for the evaluation script:

1. 2 core 4 GB memory
2. 8 core 16GB memory

On both setups the worker is picking up the submissions and passing it to the evaluation script as expected and triggering the evaluation. When running the evaluate method I can see the logs emitted by the argoverse package for some time but after a few minutes those logs stop and there are no failures from the evaluate method. I have attached the evaluate method logs that were received for last submission. If you find anything in the logs let us know.

We are looking into the issue to figure out what the exact issue is but so far we haven't noticed any issues in the setup itself. I remember we had tested some submissions for the challenge a few weeks ago. Was the evaluation script updated in meantime? If yes, could you take another look at the changes to evaluation script at your end to see if there are any issues. I'll update you once I have some findings.

[worker_logs.txt](https://github.com/Cloud-CV/EvalAI/files/5977150/worker_logs.txt)


Hi @Ram81 and @RishabhJain2018 , thank you for looking into this.

We did update our script recently, but haven't had any problems running it locally. However, t should be a bit more memory-intensive now.

The logs look pretty standard, but it looks like things start to hang about 2% of the way through computation.  It is quite strange. Would there be any other way for us to test it together?

Would this in any way be related to the following execution times?
<img width="906" alt="Screen Shot 2021-02-15 at 2 39 40 PM" src="https://user-images.githubusercontent.com/16724970/107987289-b6a7db00-6f9b-11eb-98af-30578ea8b2e3.png">
<img width="911" alt="Screen Shot 2021-02-15 at 2 39 33 PM" src="https://user-images.githubusercontent.com/16724970/107987291-b8719e80-6f9b-11eb-8042-c5473fdf1be5.png">

Hi @johnwlambert,

After some debugging I found that argoverse package is doing some multiprocessing to evaluate the submissions. During this step some processes are ending up in `defunct` state.

```
root         1  4.4  1.3 3825476 878496 ?      Ssl  05:04   1:44 python -m scripts.workers.submission_worker
root       182  1.0  1.3 3652308 854768 ?      S    05:05   0:24 python -m scripts.workers.submission_worker
root       183  1.0  0.0      0     0 ?        Z    05:05   0:24 [python] <defunct>
root       184  1.0  0.0      0     0 ?        Z    05:05   0:23 [python] <defunct>
root       185  1.0  0.0      0     0 ?        Z    05:05   0:23 [python] <defunct>
root       186  0.9  0.0      0     0 ?        Z    05:05   0:21 [python] <defunct>
root       187  0.9  0.0      0     0 ?        Z    05:05   0:22 [python] <defunct>
root       188  0.9  0.0      0     0 ?        Z    05:05   0:22 [python] <defunct>
```

In the above log `python -m scripts.workers.submission_worker` is the evaluation script process and all other processes which are in `defunct` state were created by the argoverse package during evaluation. I found [this](https://stackoverflow.com/questions/45505981/multiprocessing-produces-defunct-process) on a quick search. Is it possible for you to look at your multiprocessing implementation to see if it process pool is handled correctly.

@johnwlambert [this](https://pythonspeed.com/articles/python-multiprocessing/) blog also has more details on why multiprocessing pools get stuck and a way to use `spawn` method instead of `fork` to start processes in a multiprocessing pool to fix this issue. Let me know if we can try this change in the evaluation script.

Thanks for your help @Ram81  and @RishabhJain2018 -- we added a flag to run the eval script in a single process (instead of using a Pool) and the problem went away.