Celery worker on Kubernetes error (permission denied)
Tags: airflow, kubernetes, celery, billiard
Published: 2023-12-08 09:55:54

Hi everyone,

I am trying to setup airflow on Kubernetes but without using the helm charts, rather using simple deployment scripts. I have successfully managed to run webserver, scheduler, and triggerer and connect to each other and also to the postgreSQL db. 

Currently, trying to setup the celery worker, but I am getting some permission denied error from the **billiard** library as following:

`[2023-12-08 09:36:50,561: CRITICAL/MainProcess] Unrecoverable error: FileNotFoundError(2, 'No such file or directory')
Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.9/site-packages/celery/worker/worker.py", line 202, in start
    self.blueprint.start(self)
  File "/home/airflow/.local/lib/python3.9/site-packages/celery/bootsteps.py", line 116, in start
    step.start(parent)
  File "/home/airflow/.local/lib/python3.9/site-packages/celery/bootsteps.py", line 365, in start
    return self.obj.start()
  File "/home/airflow/.local/lib/python3.9/site-packages/celery/concurrency/base.py", line 130, in start
    self.on_start()
  File "/home/airflow/.local/lib/python3.9/site-packages/celery/concurrency/prefork.py", line 109, in on_start
    P = self._pool = Pool(processes=self.limit,
  File "/home/airflow/.local/lib/python3.9/site-packages/celery/concurrency/asynpool.py", line 464, in __init__
    super().__init__(processes, *args, **kwargs)
  File "/home/airflow/.local/lib/python3.9/site-packages/billiard/pool.py", line 1045, in __init__
    self._create_worker_process(i)
  File "/home/airflow/.local/lib/python3.9/site-packages/celery/concurrency/asynpool.py", line 482, in _create_worker_process
    return super()._create_worker_process(i)
  File "/home/airflow/.local/lib/python3.9/site-packages/billiard/pool.py", line 1141, in _create_worker_process
    on_ready_counter = self._ctx.Value('i')
  File "/home/airflow/.local/lib/python3.9/site-packages/billiard/context.py", line 179, in Value
    return Value(typecode_or_type, *args, lock=lock,
  File "/home/airflow/.local/lib/python3.9/site-packages/billiard/sharedctypes.py", line 81, in Value
    lock = ctx.RLock()
  File "/home/airflow/.local/lib/python3.9/site-packages/billiard/context.py", line 108, in RLock
    return RLock(ctx=self.get_context())
  File "/home/airflow/.local/lib/python3.9/site-packages/billiard/synchronize.py", line 206, in __init__
    SemLock.__init__(self, RECURSIVE_MUTEX, 1, 1, ctx=ctx)
  File "/home/airflow/.local/lib/python3.9/site-packages/billiard/synchronize.py", line 70, in __init__
    sl = self._semlock = _billiard.SemLock(
FileNotFoundError: [Errno 2] No such file or directory`

As there is no directory mentioned (following some github issues), I doubt that it could be with /dev/shm related directory to which I have provided access for the user running the celery worker. Has anyone had the same problem? If so, can you share some hints on possible solution to it? Note that the default behaviour of our Kubernetes is read-only, meanwhile we mount a emptyDir for the /dev/shm, in order to write-read there. Thanks in advance!