Please update Boto3 to >= 1.18.4 for aws-lambda-deploy
Asked by sergio_alba:

Hi!

The current version for Boto3 is below 1.18.4, where a ResourceConflictException error related to AWS Lambda Deploy has been fixed ([src](https://github.com/boto/boto3/blob/develop/CHANGELOG.rst#1184)). This error is related to the changes explained [here](https://aws.amazon.com/de/blogs/compute/coming-soon-expansion-of-aws-lambda-states-to-all-functions/) by the AWS team.

*** ** * ** ***

Pipeline output:

    d0f52b114f67: Pull complete

    bed928cd028b: Pull complete

    Digest: sha256:04ecf0a4530e7cdafe22a638ddcd8f63c4e308f77b1e4ea1e1dd4aa106656b00

    Status: Downloaded newer image for bitbucketpipelines/aws-lambda-deploy:1.1.0

    INFO: Using default authentication with AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY

    DEBUG: Starting new HTTPS connection (1): http://bitbucket.org:443

    DEBUG: https://bitbucket.org:443 "GET /bitbucketpipelines/official-pipes/raw/master/pipes.prod.json HTTP/1.1" 200 None

    INFO: Executing the aws-lambda-deploy pipe...

    DEBUG: Changing event name from creating-client-class.iot-data to creating-client-class.iot-data-plane

    DEBUG: Changing event name from before-call.apigateway to before-call.api-gateway

    DEBUG: Changing event name from request-created.machinelearning.Predict to request-created.machine-learning.Predict

    DEBUG: Changing event name from before-parameter-build.autoscaling.CreateLaunchConfiguration to before-parameter-build.auto-scaling.CreateLaunchConfiguration

    DEBUG: Changing event name from before-parameter-build.route53 to before-parameter-build.route-53

    DEBUG: Changing event name from http://request-created.cloudsearchdomain.Search to http://request-created.cloudsearch-domain.Search

    DEBUG: Changing event name from docs.*.autoscaling.CreateLaunchConfiguration.complete-section to docs.*.auto-scaling.CreateLaunchConfiguration.complete-section

    DEBUG: Changing event name from before-parameter-build.logs.CreateExportTask to before-parameter-build.cloudwatch-logs.CreateExportTask

    DEBUG: Changing event name from docs.*.logs.CreateExportTask.complete-section to docs.*.cloudwatch-logs.CreateExportTask.complete-section

    DEBUG: Changing event name from http://before-parameter-build.cloudsearchdomain.Search to http://before-parameter-build.cloudsearch-domain.Search

    DEBUG: Changing event name from docs.*.cloudsearchdomain.Search.complete-section to docs.*.cloudsearch-domain.Search.complete-section

    DEBUG: IMDS ENDPOINT: http://169.254.169.254/

    DEBUG: Looking for credentials via: env

    INFO: Found credentials in environment variables.

    DEBUG: Loading JSON file: /usr/local/lib/python3.9/site-packages/botocore/data/endpoints.json

    DEBUG: Event choose-service-name: calling handler <function handle_service_name_alias at 0x7fa511c20f70>

    DEBUG: Loading JSON file: /usr/local/lib/python3.9/site-packages/botocore/data/lambda/2015-03-31/service-2.json

    DEBUG: Event creating-client-class.lambda: calling handler <function add_generate_presigned_url at 0x7fa511c4d790>

    DEBUG: Setting lambda timeout as (60, 60)

    DEBUG: Loading JSON file: /usr/local/lib/python3.9/site-packages/botocore/data/_retry.json

    DEBUG: Registering retry handlers for service: lambda

    INFO: Updating Lambda function configuration.

    DEBUG: Event before-parameter-build.lambda.UpdateFunctionConfiguration: calling handler <function generate_idempotent_uuid at 0x7fa511bca670>

    DEBUG: Event before-call.lambda.UpdateFunctionConfiguration: calling handler <function inject_api_version_header_if_needed at 0x7fa511bcbee0>

    DEBUG: Making request for OperationModel(name=UpdateFunctionConfiguration) with params: {'url_path': '/2015-03-31/functions/xxxx/configuration', 'query_string': {}, 'method': 'PUT', 'headers': {'User-Agent': 'Boto3/1.17.112 Python/3.9.6 Linux/5.4.92-flatcar Botocore/1.20.112'}, 'body': b'{"Role": "arn:aws:iam::205746061435:role/Nexplayer-License-Lambda", "Handler": "index.lambdaHandler", "Description": "DEV API v2 deployment", "Timeout": 30, "MemorySize": 256, "Environment": {"Variables": {"DEBUG": "true", "S3ApiVersion": "2006-03-01", "S3BucketName": "nexplayer-license-dev", "S3Region": "us-east-1"}}, "Layers": ["arn:aws:lambda:us-east-1:205746061435:layer:nexplayer-logger:6", "arn:aws:lambda:us-east-1:205746061435:layer:nexplayer-common:16", "arn:aws:lambda:us-east-1:205746061435:layer:psl:1", "arn:aws:lambda:us-east-1:205746061435:layer:axios:2"], "Runtime": "nodejs14.x"}', 'url': '', 'context': {'client_region': 'us-east-1', 'client_config': <botocore.config.Config object at 0x7fa5110f3100>, 'has_streaming_input': False, 'auth_type': None}}

    DEBUG: Event request-created.lambda.UpdateFunctionConfiguration: calling handler <bound method RequestSigner.handler of <botocore.signers.RequestSigner object at 0x7fa5110f3070>>

    DEBUG: Event choose-signer.lambda.UpdateFunctionConfiguration: calling handler <function set_operation_specific_signer at 0x7fa511bca550>

    DEBUG: Calculating signature using v4 auth.

    DEBUG: CanonicalRequest:

    PUT

    /2015-03-31/functions/xxxxx/configuration

    host:lambda.us-east-1.amazonaws.com

    x-amz-date:20211005T075411Z

    host;x-amz-date

    5a462927d44c5d995dd83c3c680c03fa0c800043545d962072da3c82fa65e0e4

    DEBUG: StringToSign:

    AWS4-HMAC-SHA256

    20211005T075411Z

    20211005/us-east-1/lambda/aws4_request

    a188161dbb945e68d163fd50c27d0ef084134a3dcfea3d8cf3b05220d2d345c5

    DEBUG: Signature:

    df3e3ecc4ba0f1a04a8c0be2838aefa434fff06b16cd9fe60fd7add846528006

    DEBUG: Sending http request: <AWSPreparedRequest stream_output=False, method=PUT, url=https://lambda.us-east-1.amazonaws.com/2015-03-31/functions/xxxx/configuration, headers={'User-Agent': b'Boto3/1.17.112 Python/3.9.6 Linux/5.4.92-flatcar Botocore/1.20.112', 'X-Amz-Date': b'20211005T075411Z', 'Authorization': b'AWS4-HMAC-SHA256 Credential=$AWSID/20211005/us-east-1/lambda/aws4_request, SignedHeaders=host;x-amz-date, Signature=df3e3ecc4ba0f1a04a8c0be2838aefa434fff06b16cd9fe60fd7add846528006', 'Content-Length': '598'}>

    DEBUG: Certificate path: /usr/local/lib/python3.9/site-packages/certifi/cacert.pem

    DEBUG: Starting new HTTPS connection (1): http://lambda.us-east-1.amazonaws.com:443

    DEBUG: https://lambda.us-east-1.amazonaws.com:443 "PUT /2015-03-31/functions/xxxxx/configuration HTTP/1.1" 200 1825

    DEBUG: Response headers: {'Date': 'Tue, 05 Oct 2021 07:54:11 GMT', 'Content-Type': 'application/json', 'Content-Length': '1825', 'Connection': 'keep-alive', 'x-amzn-RequestId': 'f602613b-21ca-4e66-be6e-47b2f01fcbb1'}

    DEBUG: Response body:

    b'{"Description":"DEV API v2 deployment","TracingConfig":{"Mode":"PassThrough"},"VpcConfig":{"VpcId":"","SecurityGroupIds":[],"SubnetIds":[]},"SigningJobArn":null,"RevisionId":"3814806b-7ac5-4944-9c26-60cf087e72f8","LastModified":"2021-10-05T07:54:11.000+0000","FileSystemConfigs":null,"FunctionName":"xxx","Runtime":"nodejs14.x","Version":"$LATEST","PackageType":null,"LastUpdateStatus":"InProgress","Layers":","KMSKeyArn":null,"MemorySize":256,"ImageConfigResponse":null,"LastUpdateStatusReason":"The function is being created.","DeadLetterConfig":null,"Timeout":30,"Handler":"index.lambdaHandler","CodeSha256":"aoyZfk2bd+c28fwRU+sC5oUiZu5FHhyEeFwDmk6Z/tU=","Role":"arn:aws:iam::205746061435:role//***/","SigningProfileVersionArn":null,"MasterArn":null,"CodeSize":16107,"State":"Active","StateReason":null,"Environment":{"Variables":{"S3Region":"us-east-1","S3ApiVersion":"2006-03-01","S3BucketName":"***","DEBUG":"true"},"Error":null},"StateReasonCode":null,"LastUpdateStatusReasonCode":"Creating","Architectures":["x86_64"]}'

    DEBUG: Event needs-retry.lambda.UpdateFunctionConfiguration: calling handler <botocore.retryhandler.RetryHandler object at 0x7fa5110f3a60>

    DEBUG: No retry needed.

    ? Lambda configuration update succeeded.

    INFO: Updating Lambda function.

    DEBUG: Event before-parameter-build.lambda.UpdateFunctionCode: calling handler <function generate_idempotent_uuid at 0x7fa511bca670>

    DEBUG: Event before-call.lambda.UpdateFunctionCode: calling handler <function inject_api_version_header_if_needed at 0x7fa511bcbee0>

    DEBUG: Making request for OperationModel(name=UpdateFunctionCode) with params: {'url_path': '/2015-03-31/functions/***/code', 'query_string': {}, 'method': 'PUT', 'headers': {'User-Agent': 'Boto3/1.17.112 Python/3.9.6 Linux/5.4.92-flatcar Botocore/1.20.112'}, 'body': b'{"Publish": true, "ZipFile": , 'url': '', 'context': {'client_region': 'us-east-1', 'client_config': <botocore.config.Config object at 0x7fa5110f3100>, 'has_streaming_input': False, 'auth_type': None}}

    DEBUG: Event request-created.lambda.UpdateFunctionCode: calling handler <bound method RequestSigner.handler of <botocore.signers.RequestSigner object at 0x7fa5110f3070>>

    DEBUG: Event choose-signer.lambda.UpdateFunctionCode: calling handler <function set_operation_specific_signer at 0x7fa511bca550>

    DEBUG: Calculating signature using v4 auth.

    DEBUG: CanonicalRequest:

    PUT

    /2015-03-31/functions/****/code

    host:lambda.us-east-1.amazonaws.com

    x-amz-date:20211005T075411Z

    host;x-amz-date

    43ec3d19fdba9634a61fb714d9d2b0c1a5b14a3f4f96fe1dffcf2e52b97b5706

    DEBUG: StringToSign:

    AWS4-HMAC-SHA256

    20211005T075411Z

    20211005/us-east-1/lambda/aws4_request

    ac68570bf41afea71521dac5989439e6b38c726adb91f1ff2fb26a62b1115814

    DEBUG: Signature:

    49ebcda9365ecf57d7ae62b1b9bff742c6a81212eea2261f1397419830860b10

    DEBUG: Sending http request: <AWSPreparedRequest stream_output=False, method=PUT, url=https://lambda.us-east-1.amazonaws.com/2015-03-31/functions/****/code, headers={'User-Agent': b'Boto3/1.17.112 Python/3.9.6 Linux/5.4.92-flatcar Botocore/1.20.112', 'X-Amz-Date': b'20211005T075411Z', 'Authorization': b'AWS4-HMAC-SHA256 Credential=$AWSID/20211005/us-east-1/lambda/aws4_request, SignedHeaders=host;x-amz-date, Signature=49ebcda9365ecf57d7ae62b1b9bff742c6a81212eea2261f1397419830860b10', 'Content-Length': '23060'}>

    DEBUG: Certificate path: /usr/local/lib/python3.9/site-packages/certifi/cacert.pem

    DEBUG: https://lambda.us-east-1.amazonaws.com:443 "PUT /2015-03-31/functions/xxxx/code HTTP/1.1" 409 180

    DEBUG: Response headers: {'Date': 'Tue, 05 Oct 2021 07:54:11 GMT', 'Content-Type': 'application/json', 'Content-Length': '180', 'Connection': 'keep-alive', 'x-amzn-RequestId': '07322f5c-ebe9-43f8-9f4d-da4154f44c4f', 'x-amzn-ErrorType': 'ResourceConflictException'}

    DEBUG: Response body:

    b'{"Type":"User","message":"The operation cannot be performed at this time. An update is in progress for resource: arn:aws:lambda:us-east-1:****

    DEBUG: Response headers: {'Date': 'Tue, 05 Oct 2021 07:54:11 GMT', 'Content-Type': 'application/json', 'Content-Length': '180', 'Connection': 'keep-alive', 'x-amzn-RequestId': '07322f5c-ebe9-43f8-9f4d-da4154f44c4f', 'x-amzn-ErrorType': 'ResourceConflictException'}

    DEBUG: Response body:

    b'{"Type":"User","message":"The operation cannot be performed at this time. An update is in progress for resource: arn:aws:lambda:us-east-1:****

    DEBUG: Event needs-retry.lambda.UpdateFunctionCode: calling handler <botocore.retryhandler.RetryHandler object at 0x7fa5110f3a60>

    DEBUG: No retry needed.

    ? Failed to update Lambda function code. Error: An error occurred (ResourceConflictException) when calling the UpdateFunctionCode operation: The operation cannot be performed at this time. An update is in progress for resource: arn:aws:lambda:us-east-1:xxxx


Answers:
author: Oleksandr Kyrdan
body: Hi [@sergio_alba](/t5/user/viewprofilepage/user-id/4584673)

Thank you for your question!

We'll investigate the issue and notify you.

Best regards,  
Oleksandr Kyrdan

comments: author: sergio_alba
body: Hi Oleksandr,

Many thanks for your response! Any updates on this will be greatly appreciated since we heavily depend on the pipe for our AWS Lambda Deployments.

Best regards,

Sergio.


author: Oleksandr Kyrdan
body: Hi [@sergio_alba](/t5/user/viewprofilepage/user-id/4584673) ,

Good news we have released updated version of the [aws-lambda-deploy](https://bitbucket.org/atlassian/aws-lambda-deploy) pipe:

* boto3 bumped to the latest 1.18.\* version

```
script:  - pipe: atlassian/aws-lambda-deploy:1.2.0    variables:      AWS_ACCESS_KEY_ID: $AWS_ACCESS_KEY_ID      AWS_SECRET_ACCESS_KEY: $AWS_SECRET_ACCESS_KEY      AWS_DEFAULT_REGION: $AWS_DEFAULT_REGION      FUNCTION_NAME: 'my-lambda-function'      COMMAND: 'update'      ZIP_FILE: 'my-function-code.zip'
```

It would be nice if you could provide us with your feedback about updated pipe.

Cheers,  
Oleksandr Kyrdan


author: sergio_alba
body: Hi [@Oleksandr Kyrdan](/t5/user/viewprofilepage/user-id/2885705),

Thanks a lot for the update! I have tested it on a test function and the pipe works.

Best regards and thanks for all the good work,

Sergio.

EDIT:

Hi again [@Oleksandr Kyrdan](/t5/user/viewprofilepage/user-id/2885705) ,

I tried running the pipeline on our dev ver. stack with no luck, although the pipe version is correct I keep getting the error message:

```
?Status: Downloaded newer image for bitbucketpipelines/aws-lambda-deploy:1.2.0
[...]
INFO: Updating Lambda function configuration.
? Lambda configuration update succeeded.
INFO: Updating Lambda function.
? Failed to update Lambda function code. Error: An error occurred (ResourceConflictException) when calling the UpdateFunctionCode operation: The operation cannot be performed at this time. An update is in progress for resource: arn:aws:lambda:us-east-1:xxxxx:function:xxxx
```

Any ideas on why we keep getting this error?

Best regards,

Sergio.


author: Alejandro Alfonso Fern?ndez
body: Hi!

Same problem in our recipe: the function updates right, but the logs returns the same error [@sergio_alba](/t5/user/viewprofilepage/user-id/4584673) reports

Best regards


author: sergio_alba
body: Hi!

Any updates on this issue?

Thanks,


author: Igor Stoyanov
body: [@sergio_alba](/t5/user/viewprofilepage/user-id/4584673) Hi. Please provide configuration of your pipe. Seems like you use it with FUNCTION_CONFIGURATION and despite of the fact of this log message:

```
? Lambda configuration update succeeded.
```

the state of the lambda is not equal:

    {
        "State": "Active",    "LastUpdateStatus": "Successful"
    }

so *UpdateFunctionCode*operation fails.

Can you confirm that you use in your pipe configuration FUNCTION_CONFIGURATION parameter and check if all is ok with only update function code without updating function configuration ?

From our side we will try to reproduce and fix this error.

Regards, Igor.


author: sergio_alba
body: [@Igor Stoyanov](/t5/user/viewprofilepage/user-id/4550492) Hi.

#1 Function logs (Ok):

\[...\]  
Status: Downloaded newer image for bitbucketpipelines/aws-lambda-deploy:1.2.0

INFO: Using default authentication with AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY

INFO: Executing the aws-lambda-deploy pipe...

INFO: Found credentials in environment variables.

INFO: Updating Lambda function.

#2 Function logs (Failed):

\[...\]  
Digest: sha256:5b74f29da9b7d58ea23a6cadea54d7a3959556e0e0dd0653b31b6dae6e4828df

Status: Downloaded newer image for bitbucketpipelines/aws-lambda-deploy:1.2.0

INFO: Using default authentication with AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY

INFO: Executing the aws-lambda-deploy pipe...

INFO: Found credentials in environment variables.

INFO: Updating Lambda function configuration.

? Lambda configuration update succeeded.

INFO: Updating Lambda function.

? Failed to update Lambda function code. Error: An error occurred (ResourceConflictException) when calling the UpdateFunctionCode operation: The operation cannot be performed at this time. An update is in progress for resource: arn:aws:lambda:us-east-1:205746061435:function:check-user-license

I have removed the FUNCTION_CONFIGURATION parameter on the first function deployment and it indeed worked. I left this parameter on the second function of the pipeline, which did indeed fail.

Configuration is as follows (redacted zipping of function files and function names):

- step:

name: Deploy \[function-name\]

script:

- pipe: atlassian/aws-lambda-deploy:1.2.0

variables:

AWS_ACCESS_KEY_ID: ${AWSID}

AWS_SECRET_ACCESS_KEY: ${AWSKEY}

AWS_DEFAULT_REGION: 'us-east-1'

FUNCTION_NAME: 'function-name'

COMMAND: 'update'

ZIP_FILE: 'function-name/code.zip'

# FUNCTION_CONFIGURATION: 'function-name/dev_config.json'

# Delete dir, otherwise it fails

- rm /opt/atlassian/pipelines/agent/build/.bitbucket/pipelines/generated/pipeline/pipes/aws-lambda-deploy-env

- step:

name: Deploy \[function-name\]

script:

- pipe: atlassian/aws-lambda-deploy:1.2.0

variables:

AWS_ACCESS_KEY_ID: ${AWSID}

AWS_SECRET_ACCESS_KEY: ${AWSKEY}

AWS_DEFAULT_REGION: 'us-east-1'

FUNCTION_NAME: 'check-user-license'

COMMAND: 'update'

ZIP_FILE: 'function-name/code.zip'

FUNCTION_CONFIGURATION: 'function-name/dev_config.json'

- rm /opt/atlassian/pipelines/agent/build/.bitbucket/pipelines/generated/pipeline/pipes/aws-lambda-deploy-env


author: Igor Stoyanov
body: [@sergio_alba](/t5/user/viewprofilepage/user-id/4584673) Hi.

Thanks for your response. As i see the problem is with lambda state between configuration and code update. We will provide a fix in near future.

Regards, Igor.


author: Igor Stoyanov
body: [@sergio_alba](/t5/user/viewprofilepage/user-id/4584673) hi. We released new version of aws-lambda-deploy pipe with implemented waiter. Now you could do lambda update configuration and update code with the custom WAIT_INTERVAL:

```
script:  - pipe: atlassian/aws-lambda-deploy:1.3.0    variables:      AWS_ACCESS_KEY_ID: $AWS_ACCESS_KEY_ID      AWS_SECRET_ACCESS_KEY: $AWS_SECRET_ACCESS_KEY      AWS_DEFAULT_REGION: $AWS_DEFAULT_REGION      FUNCTION_NAME: 'my-lambda-function'      COMMAND: 'update'      ZIP_FILE: 'my-function-code.zip'      FUNCTION_CONFIGURATION: 'new_function_configuration.json'      WAIT_INTERVAL: '15'
```

or with default WAIT_INTERVAL=5 sec if you don't want to provide your custom value.

Let us know if the new version of pipe fixes your problem.

Regards, Igor.


Tags: bitbucket-pipelines

aws

cloud

lambda