Repository: pytorch
Issue: Gunicorn preload flag not working with PyTorch library
Labels: module: nn,triaged

Hello, 
We have been developing a FastAPI application where we use some external libraries to perform some NLP tasks, such as tokenization. On top of this, we are launching the service with Gunicorn so that we can parallelize the requests.
However, we are having difficulties using [Stanza](https://stanfordnlp.github.io/stanza/) with Gunicorn’s preload flag active.
It is a requirement to use this flag because since Stanza models can be large, we want the models to be loaded only once, in the Gunicorn master process. This way, Gunicorn workers can access the models that were previously loaded in the master process.

The difficulties that we are facing resumes on the fact that Gunicorn workers hang when trying to make an inference over a given model (that was loaded initially by the master process).

We’ve done some research and debugging but we weren’t able to find a solution. However, we noticed that the worker hangs when the code reaches the prediction step on PyTorch.
Although we are talking about Stanza, this problem also occurred with [Sentence Transformers library](https://github.com/UKPLab/sentence-transformers). And both of them are using the PyTorch library. 

Following, I’ll present more details:

**Environment:**
```
FastApi version: 0.54.2
Gunicorn version: 20.0.4
Uvicorn version: 0.12.3
Python version: 3.7
Stanza version: 1.1.1
OS: macOS Catalina 10.15.6
```
**Steps executed:**

- Gunicorn command:
```
gunicorn --workers 1 --worker-class uvicorn.workers.UvicornWorker --max-requests=0 --max-requests-jitter=0 --timeout=120 --keep-alive=2 \
     --log-level=info --access-logfile - --preload -b 0.0.0.0:8010 my_app:app
```

- The code that will run before launching the workers
```
def initialize_application() -> None:
     ...
     model = stanza.Pipeline(
                lang=cls._TOKENIZER_MODEL_LANGUAGES[language],
                package=cls._MODEL_TYPE[language],
                processors=cls._TOKENIZER_MODEL,
                tokenize_no_ssplit=True,
            )

```
This way, the model can be loaded only once, in the master process. 
Once the required workers are launched, they should have access to the previous model, without having to load it by themselves (saving computational resources).

The problem happens when we receive a request that will make use of the model that was initially loaded. The worker that will be responsible for handling the request, won’t be able to use the model for inference. As so, the worker will be hanged until the timeout occurs.

After analyzing the code and debugging it, we reached the following step until the code stopped working:

1. Our code has a call to the `process()` method, `class Pipeline`, on the core.py file of Stanza.
2. That line calls the specific `process()` method, in this case from the tokenize_process.py, `class TokenizeProcessor`
3. Which calls the PyTorch code, `output_predictions()` method, from the utils.py
4. After some steps, it reaches the model.py file still in PyTorch, `class Tokenizer(nn.Module)`, `forward(self, x, feats) method`, in the following line: `nontok = F.logsigmoid(-tok0)`. It seems that this line is calling some C++ code where we didn’t investigate any further.

Of course, if we remove the --preload flag, everything will run smoothly. Removing it is something that we want to avoid because of the added computational resources that will be necessary (the models will be duplicated in every worker).

We looked through several other issues that could be related to this one, such as:
https://github.com/benoitc/gunicorn/issues/2157
https://github.com/tiangolo/fastapi/issues/2425
https://github.com/tiangolo/fastapi/issues/596
https://github.com/benoitc/gunicorn/issues/2124
and others...

After trying multiple solutions, it wasn’t possible to solve the issue. Do you have any suggestions to handle this? Or other tests that I can perform to give you more information?

Thanks in advance.

cc @albanD @mruberry @jbschlosser @walterddr @mikaylagawarecki @zhangguanheng66