Why does serverless-deploy pipe use serverless-python-requirements without poetry installed?
Asked by Sam Mahr:

I have a serverless app that uses Poetry for package management.

I am using bitbucketpipelines/serverless-deploy:1.5.0 to deploy my serverless app. When I try to deploy, I get this error:

```
Deploying ServerlessAppTemplate to stage dev (us-east-2)
Generating requirements.txt from "pyproject.toml"
? Stack ServerlessAppTemplate-dev failed to deploy (0s)
Environment: linux, node 16.20.2, framework 3.38.0, plugin 7.2.0, SDK 4.5.1
Credentials: Local, environment variables
Docs: docs.serverless.com
Support: forum.serverless.com
Bugs: github.com/serverless/serverless/issues
Error:
Error: spawn poetry ENOENT
at Process.ChildProcess._handle.onexit (node:internal/child_process:285:19)
at onErrorNT (node:internal/child_process:485:16)
at processTicksAndRejections (node:internal/process/task_queues:83:21)
```

Why is serverless-python-requirements part of the Docker Image, which supports Poetry projects, but poetry is not installed in the image?

I've tried installing poetry in a pre execution script, and I still get the same error. Can the serverless-deploy pipe's Dockerfile include the installation of Poetry?

Thanks


Answers:
author: Oleksandr Kyrdan
body: [@Sam Mahr](/t5/user/viewprofilepage/user-id/2787874)

Thank you for your question!

The pipe's base docker image contains only base dependencies to keep the image thin.

There are many variety of dependencies all around the users projects.

If you need extra dependencies according to your project, pipe supports preexecution script.

Additionally, check if the version of requirements and runtime corresponds to platform dependency version in Dockerfile.

Also, could you provide your bitbucket-pipelines configuration, serverless configuration and preexecution script.

Best regards,  
Oleksandr Kyrdan

comments: author: Sam Mahr
body: Thanks, [@Oleksandr Kyrdan](/t5/user/viewprofilepage/user-id/2885705). Here's the files

```
apt-get update
apt-get install -y git

# START SSH setup
INJECTED_SSH_CONFIG_DIR="/opt/atlassian/pipelines/agent/ssh"
# The default ssh key with open perms readable by alt uids
IDENTITY_FILE="${INJECTED_SSH_CONFIG_DIR}/id_rsa_tmp"
# The default known_hosts file
KNOWN_HOSTS_FILE="${INJECTED_SSH_CONFIG_DIR}/known_hosts"

mkdir -p ~/.ssh || debug "adding ssh keys to existing ~/.ssh"
touch ~/.ssh/authorized_keys

# If given, use SSH_KEY, otherwise check if the default is configured and use it
if [ "${SSH_KEY}" != "" ]; then
   debug "Using passed SSH_KEY"
   (umask  077 ; echo ${SSH_KEY} | base64 -d > ~/.ssh/pipelines_id)
elif [ ! -f ${IDENTITY_FILE} ]; then
   error "No default SSH key configured in Pipelines."
   exit 1
else
   debug "Using default ssh key"
   cp ${IDENTITY_FILE} ~/.ssh/pipelines_id
fi

if [ ! -f ${KNOWN_HOSTS_FILE} ]; then
    error "No SSH known_hosts configured in Pipelines."
    exit 2
fi

cat ${KNOWN_HOSTS_FILE} >> ~/.ssh/known_hosts
if [ -f ~/.ssh/config ]; then
    debug "Appending to existing ~/.ssh/config file"
fi
echo "IdentityFile ~/.ssh/pipelines_id" >> ~/.ssh/config
chmod -R go-rwx ~/.ssh/
# END SSH setup

curl -sSL https://install.python-poetry.org | python3 -
export PATH="$HOME/.local/bin:$PATH"
poetry --version
poetry config http-basic.artifact aws "$TOKEN"

npm install --save -g serverless-step-functions
```

pre execution script

*** ** * ** ***

```
service: ServerlessAppTemplate
frameworkVersion: ^3.38.0
package:
  individually: true
  patterns:
    - '!config/**'
    - '!lambda_functions/**'
    - '!*.json'
    - '!*.md'
    - '!*.txt'
    - '!.gitignore'
    - '!pytest.ini'
    - '!tests/**'
    - '!*.yml'
    - '!*.yaml'
    - '!Makefile'
    - '!poetry.lock'
    - '!pyproject.toml'

provider:
  name: aws
  runtime: python3.10
  httpApi:
    cors: true
  stage: ${opt:stage, "dev"}
  region: ${opt:region, "us-east-2"}
  deploymentBucket: 
    name: [redacted]-${self:provider.region}
  deploymentPrefix: Serverless
  environment:
    REGION: ${self:provider.region}
    STAGE: ${self:provider.stage}
    POWERTOOLS_SERVICE_NAME: template
    LOG_LEVEL: INFO

functions: ${file(config/functions.yml)}

resources: ${file(config/resources.yml)}

stepFunctions: ${file(config/state_machines.yml)}

plugins:
  - serverless-python-requirements
  - serverless-step-functions

custom:
  powertoolsLayer: arn:aws:lambda:${opt:region}:017000801446:layer:AWSLambdaPowertoolsPythonV2:67
  pythonRequirements:
    layer: true
    slim: true
    strip: false
    usePoetry: true
```

Serverless config with explicit usePoetry

*** ** * ** ***

```
image: python:3.10

definitions:
  caches:
    poetry-deps:
      key:
        files:
          - poetry.lock
      path: .venv
  steps:
    - step: &get-token-step
        name: Get CodeArtifact Token
        caches:
          - pip
          - poetry-deps
        image: atlassian/pipelines-awscli
        script:
          - export AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID
          - export AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY
          - export AWS_DEFAULT_REGION=us-east-2
          - TOKEN=$(aws codeartifact get-authorization-token --region us-east-2 --domain [redacted] --query authorizationToken --output text)
          - echo "export TOKEN=$TOKEN" >> set_env.sh
        artifacts:
          - set_env.sh
    - step: &audit-step
        name: Audit Packages
        caches:
          - docker
          - pip
          - poetry-deps
        script:
          - source set_env.sh
          - pipe: atlassian/poetry-cli-setup:0.2.0
          - POETRY_VIRTUALENVS_IN_PROJECT=true ./setup-poetry.sh
          - poetry config http-basic.artifact aws $TOKEN
          - poetry install
          - poetry self add poetry-audit-plugin
          - poetry audit
    - step: &test-step
        name: Test
        caches:
          - docker
          - pip
          - poetry-deps
        script:
          - source set_env.sh
          - pipe: atlassian/poetry-cli-setup:0.2.0
          - POETRY_VIRTUALENVS_IN_PROJECT=true ./setup-poetry.sh
          - poetry config http-basic.artifact aws $TOKEN
          - poetry install
          - poetry run python3 -m pytest --cov=./ --cov-report xml
    - step: &lint-step
        name: Lint code
        caches:
          - docker
          - pip
          - poetry-deps
        script:
          - source set_env.sh
          - pipe: atlassian/poetry-cli-setup:0.2.0
          - POETRY_VIRTUALENVS_IN_PROJECT=true ./setup-poetry.sh
          - poetry config http-basic.artifact aws $TOKEN
          - poetry install
          - poetry run ruff check

    - step: &deploy-functions-step
        name: Deploy Functions
        caches:
          - docker
          - node
          - pip
        script:
          - &deploy-functions-script
            pipe: atlassian/serverless-deploy:1.5.0
            variables: &deploy-functions-variables
              AWS_ACCESS_KEY_ID: $AWS_ACCESS_KEY_ID
              AWS_SECRET_ACCESS_KEY: $AWS_SECRET_ACCESS_KEY
              EXTRA_ARGS: "-s $STAGE --region $REGION"
              DEBUG: "true"
              TOKEN: $TOKEN
              PRE_EXECUTION_SCRIPT: './pipelines_pre_execution.sh'

pipelines:
  default:
    - step: *get-token-step
    - parallel:
        - step: *audit-step
        - step: *test-step
        - step: *lint-step

  branches:
    main:
      - step: *get-token-step
      - parallel:
          - step: *audit-step
          - step: *test-step
          - step: *lint-step
      - step:
          <<: *deploy-functions-step
          name: Deploy to stage Dev, region us-east-2
          deployment: dev-us-east-2
          script:
            - source set_env.sh
            - <<: *deploy-functions-script
              variables:
                <<: *deploy-functions-variables
  tags:
    'v*':
      - step: *get-token-step
      - parallel:
          - step: *audit-step
          - step: *test-step
          - step: *lint-step
      - step:
          <<: *deploy-functions-step
          name: Deploy to stage Prod, region us-east-2
          deployment: prod-us-east-2
          trigger: manual
          script:
            - source set_env.sh
            - <<: *deploy-functions-script
              variables:
                <<: *deploy-functions-variables
```

pipelines file

*** ** * ** ***

```
image: python:3.10

definitions:
  caches:
    poetry-deps:
      key:
        files:
          - poetry.lock
      path: .venv
  steps:
    - step: &get-token-step
        name: Get CodeArtifact Token
        caches:
          - pip
          - poetry-deps
        image: atlassian/pipelines-awscli
        script:
          - export AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID
          - export AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY
          - export AWS_DEFAULT_REGION=us-east-2
          - TOKEN=$(aws codeartifact get-authorization-token --region us-east-2 --domain patronpoint-artifact --query authorizationToken --output text)
          - echo "export TOKEN=$TOKEN" >> set_env.sh
        artifacts:
          - set_env.sh
    - step: &audit-step
        name: Audit Packages
        caches:
          - docker
          - pip
          - poetry-deps
        script:
          - source set_env.sh
          - pipe: atlassian/poetry-cli-setup:0.2.0
          - POETRY_VIRTUALENVS_IN_PROJECT=true ./setup-poetry.sh
          - poetry config http-basic.artifact aws $TOKEN
          - poetry install
          - poetry self add poetry-audit-plugin
          - poetry audit
    - step: &test-step
        name: Test
        caches:
          - docker
          - pip
          - poetry-deps
        script:
          - source set_env.sh
          - pipe: atlassian/poetry-cli-setup:0.2.0
          - POETRY_VIRTUALENVS_IN_PROJECT=true ./setup-poetry.sh
          - poetry config http-basic.artifact aws $TOKEN
          - poetry install
          - poetry run python3 -m pytest --cov=./ --cov-report xml
          # Install deepsource CLI
          # - curl https://deepsource.io/cli | sh
          # From the root directory, run the report coverage command
          # upload coverage report to deepsource
          # - ./bin/deepsource report --analyzer test-coverage --key python --value-file ./coverage.xml
    - step: &lint-step
        name: Lint code
        caches:
          - docker
          - pip
          - poetry-deps
        script:
          - source set_env.sh
          - pipe: atlassian/poetry-cli-setup:0.2.0
          - POETRY_VIRTUALENVS_IN_PROJECT=true ./setup-poetry.sh
          - poetry config http-basic.artifact aws $TOKEN
          - poetry install
          - poetry run ruff check

    - step: &deploy-functions-step
        name: Deploy Functions
        caches:
          - docker
          - node
          - pip
        script:
          - &deploy-functions-script
            pipe: atlassian/serverless-deploy:1.5.0
            variables: &deploy-functions-variables
              AWS_ACCESS_KEY_ID: $AWS_ACCESS_KEY_ID
              AWS_SECRET_ACCESS_KEY: $AWS_SECRET_ACCESS_KEY
              EXTRA_ARGS: "-s $STAGE --region $REGION"
              DEBUG: "true"
              TOKEN: $TOKEN
              PRE_EXECUTION_SCRIPT: './pipelines_pre_execution.sh'

pipelines:
  default:
    - step: *get-token-step
    - parallel:
        - step: *audit-step
        - step: *test-step
        - step: *lint-step

  branches:
    main:
      - step: *get-token-step
      - parallel:
          - step: *audit-step
          - step: *test-step
          - step: *lint-step
      - step:
          name: Deploy to stage Dev, region us-east-2
          caches:
            - docker
            - node
            - pip
          deployment: dev-us-east-2
          script:
            - source set_env.sh
            - pipe: atlassian/serverless-deploy:1.5.0
              variables:
                AWS_ACCESS_KEY_ID: $AWS_ACCESS_KEY_ID
                AWS_SECRET_ACCESS_KEY: $AWS_SECRET_ACCESS_KEY
                EXTRA_ARGS: "-s $STAGE --region $REGION"
                DEBUG: "true"
                TOKEN: $TOKEN
                PRE_EXECUTION_SCRIPT: './pipelines_pre_execution.sh'
```

Same issue with this pipelines file as well

*** ** * ** ***

```
Deploying ServerlessAppTemplate to stage dev (us-east-2)
Generating requirements.txt from "pyproject.toml"
? Stack ServerlessAppTemplate-dev failed to deploy (0s)
Environment: linux, node 16.20.2, framework 3.38.0, plugin 7.2.0, SDK 4.5.1
Credentials: Local, environment variables
Docs: docs.serverless.com
Support: forum.serverless.com
Bugs: github.com/serverless/serverless/issues
Error:
Error: spawn poetry ENOENT
at Process.ChildProcess._handle.onexit (node:internal/child_process:285:19)
at onErrorNT (node:internal/child_process:485:16)
at processTicksAndRejections (node:internal/process/task_queues:83:21)
? Deployment failed :(
```


author: Sam Mahr
body: [@Oleksandr Kyrdan](/t5/user/viewprofilepage/user-id/2885705) Following up


author: Oleksandr Kyrdan
body: Hi [@Sam Mahr](/t5/user/viewprofilepage/user-id/2787874)

Could you try the new dev version of the pipe with Poetry in the pipe's base docker image before the release and share feedback:

```
- pipe: docker://bitbucketpipelines/serverless-deploy:1.5.0.163-dev
```


author: Sam Mahr
body: ```
Retrieving Poetry metadata
The latest version (1.8.3) is already installed.
Poetry (version 1.8.3)
eyJ2ZXIiOjEsImlzdSI6MTcxNTgyMjM2MiwiZW5jIjoiQTEyOEdDTSIsInRhZyI6IjQ1bzVrZ2J4b3BleXNjdU04TW9Qa0EiLCJleHAiOjE3MTU4NjU1NjIsImFsZyI6IkExMjhHQ01LVyIsIml2IjoiSXVveEsxT1dpREtyNW1qRiJ9.6uRiS-LVWp3B2eNfDi31LA.DT-uf1zhaXMHhYFJ.iB0zmDgkc0ykdf3bjA9iQAJL_5Xj9C2_Td1Oppes2xF9Krp6iTK9ffoYf5oqduOC8QtMMpET_4svNkXvLnfoi2iTBJW_W76j2rsWiEw-nPueM0-ff_J0ptZ4nc3DOJGYqfW6vtsfnlBkL3MfyyqO4ts7k01IFx7SURxHP8eTGXHFlRATtAwTgWNbwv4W4igCzirxYYNqcWUfcRkQthZrRKypOd1jMJ20_wJLpkwD5wPZl9BTWgWHZxTy9xXdHLXjldWK24CLe2X62NcpKTHcWol23BrTrecD4a8EqvAmjxURyaLrbsVI_NVQmrSX26z58WKpQ2x3DcVdDKvwUC5HcmhOUrD5-AF-4c-CaUNO7qVurjzCvmcNFCL5n0usSGdmILHR0864sf7E_INYaozyt8-BvMsyCHI154teaDCYUIoSNa7bMQRWwnBZLddpQijyGT5IYUmb2i6gVyH3EmV34dgVqksI9yDngrI2Ea5Lopasb-WBYliPK3srOZF4r7xTaS_vG-SMjAXTI1X_o9-WjhyYK4k1SrMrXMu3z_fFhcVeFI_pbDXBE13UaL7swr9RwxAjuS7AEtOnFuxG1PnsonrcH508sfnXJEuI0-gx8nOOfKrtkY6ZG5ed2oV4maMhGhX2inIaBwPF6L5etz4Nk0t2NEqJJ-AVo_gnGup6vIu8JhRVk1HYBjoHjTPKW-6eOlyluQ_l6ElwoAdJm3H3RoEGn8mbTH4r9ol8HsBW5KLJg0D5ccwBfR3EQ40r3EmVmp0V3bJGlIRYgTU_H1hGI6gxqqKAZHtNzY6ttTd-bJzQCtPttMW1bwCb5LH68LuDKtSKQCZvr4X_aAcrButnk4svVD2nCIo31hP9qSVEBo_KXsMgaIYuXDiX10cz7zGaJhTY4Wi32_Rcd_5jNYXSgukqKio.befgFQ-ZgjR0mZZnWdkYVQ
added 570 packages in 21s
113 packages are looking for funding
run `npm fund` for details

INFO: Deploying your serverless application...
DEBUG: serverless deploy -c serverless.yml -s dev --region us-east-2 --verbose

Deploying ServerlessAppTemplate to stage dev (us-east-2)
Generating requirements.txt from "pyproject.toml"
Parsed requirements.txt from pyproject.toml in /opt/atlassian/pipelines/agent/build/.serverless/requirements.txt
Installing requirements from "/root/.cache/serverless-python-requirements/1f0dd62c2bede82c87a91b6c441f0d0940af59d12684e386817a7ff2757d10a6_x86_64_slspyc/requirements.txt"
Using download cache directory /root/.cache/serverless-python-requirements/downloadCacheslspyc
? Stack ServerlessAppTemplate-dev failed to deploy (1s)
Environment: linux, node 20.13.1, framework 3.38.0, plugin 7.2.3, SDK 4.5.1
Credentials: Local, environment variables
Docs: docs.serverless.com
Support: forum.serverless.com
Bugs: github.com/serverless/serverless/issues
Error:
Error: spawn python3.10 ENOENT
at ChildProcess._handle.onexit (node:internal/child_process:286:19)
at onErrorNT (node:internal/child_process:484:16)
at process.processTicksAndRejections (node:internal/process/task_queues:82:21)
? Deployment failed :(
```

Hmm, it looks like there's still an issue.

The pre execution script tries to install poetry, but it's already installed in this docker image, but still getting this issue: "Error: spawn python3.10 ENOENT"

[@Oleksandr Kyrdan](/t5/user/viewprofilepage/user-id/2885705)


author: Oleksandr Kyrdan
body: [@Sam Mahr](/t5/user/viewprofilepage/user-id/2787874) please, pay attention that pipe's docker image based on **python:3.11-slim**. Looks like you try to spawn python3.10 instead.


author: Sam Mahr
body: Ah, I did not look at the image. I've changed everything to 3.11 and now it is working and deployed! [@Oleksandr Kyrdan](/t5/user/viewprofilepage/user-id/2885705)


author: Oleksandr Kyrdan
body: Nice! Thanks for your feedback! ?


author: Oleksandr Kyrdan
body: Hi [@Sam Mahr](/t5/user/viewprofilepage/user-id/2787874)

A new version of [serverless-deploy pipe](https://bitbucket.org/atlassian/serverless-deploy/src/2.0.0/) is released:

```
- pipe: atlassian/serverless-deploy:2.0.0
```

changes:

2.0.0

* major: Breaking changes. Refactored pipe to support better auth logic and major providers AWS, GCP, AZURE.

* minor: Add OIDC support for AWS provider. Add support for AWS_OIDC_ROLE_ARN variable.

* minor: Add Poetry to the pipe's base docker image as an alternative tool for dependency management and packaging in Python.

* minor: Improved support for Azure cloud. Add serverless-azure-functions to the default environment.

* minor: Update base NodeJS version to nodejs20.

* minor: Update pipe's base docker image to python:3.11-slim.

* patch: Internal maintenance: bump project's dependencies to the latest.

Best regards,  
Oleksandr Kyrdan


Tags: bitbucket-cloud

bitbucket-pipelines

cloud

pipes

serverless