aws-s3-deploy 1.0 break upload (not happening in 0.5.0)
Asked by Leonardo Medici:

We have a pipeline that send our website build to a S3 bucket to serve files through CloudFront.

Today we tried to update our pipeline script to aws-s3-deploy 1.0 but this break the pipeline without a meaninful error.

Reverting to 0.5.0 fix the issue.

Here there is the interesting part of our pipeline script:

     develop:      - step:          name: Build          caches:            - node          script:            - npm install            - npm run build:devel          artifacts:            - home/**      - step:          name: Deploy to S3          deployment: Development          script:            - pipe: atlassian/aws-s3-deploy:1.0.0              variables:                S3_BUCKET: ${S3_BUCKET}/home                LOCAL_PATH: 'home'                ACL: 'public-read'                EXTRA_ARGS: '--exclude=*.html'            - pipe: atlassian/aws-s3-deploy:1.0.0              variables:                S3_BUCKET: ${S3_BUCKET}/home                LOCAL_PATH: 'home'                ACL: 'public-read'                CACHE_CONTROL: 'max-age=600'                EXTRA_ARGS: '--exclude=* --include=*.html'            - pipe: atlassian/aws-s3-deploy:1.0.0              variables:                S3_BUCKET: ${S3_BUCKET}                LOCAL_PATH: 'home'                ACL: 'public-read'                CACHE_CONTROL: 'max-age=600'                EXTRA_ARGS: '--exclude=* --include=index.html'

And the IAM policy attached to the user on AWS:

    {    "Version": "2012-10-17",    "Statement": [        {            "Effect": "Allow",            "Action": [                "s3:ListBucket"            ],            "Resource": [                "arn:aws:s3:::${S3_BUCKET}"            ]        },        {            "Effect": "Allow",            "Action": [                "s3:GetObject",                "s3:GetObjectAcl",                "s3:GetObjectTagging",                "s3:HeadObject",                "s3:PutObject",                "s3:PutObjectTagging",                "s3:PutObjectAcl",                "s3:DeleteObject"            ],            "Resource": [                "arn:aws:s3:::${S3_BUCKET}/*"            ]        }    ]}

Here (on Dropbox due to post characters limit) there is a full log of the upload step that fails with DEBUG: 'true' option set:  

<https://www.dropbox.com/s/aogfbqw86vy3su2/pipelineLog-3.txt?dl=0>


Answers:
author: Oleksandr Kyrdan
body: Hi [@Leonardo Medici](/t5/user/viewprofilepage/user-id/4457604) ,

Thank you for your question!

With version 1.0.0 we migrate aws-cli to v2. Pipe based on docker image amazon/aws-cli:2.2.13. This version is not officially released yet, but thank you for sharing your case.

Good explanation. If you ok we'll ask you some more details soon about this case.

We're investigating this issue and will notify you.

Best regards,  
Oleksandr Kyrdan

comments: author: Leonardo Medici
body: Of course, feel free to contact me if you need additional information.


author: Oleksandr Kyrdan
body: Looks like this issue relates to the changes in [behavior between AWS CLI version 1 and AWS CLI version 2](https://docs.aws.amazon.com/cli/latest/userguide/cliv2-migration.html)

One of the reason could be [locale](https://docs.aws.amazon.com/cli/latest/userguide/cliv2-migration.html#cliv2-migration-encodingenvvar):

**AWS CLI version 2 now uses environment variable to set text file encoding**

By default, text files use the same encoding as the installed locale. To set encoding for text files to be different from the locale, use the `AWS_CLI_FILE_ENCODING` environment variable. The below example sets the CLI to open text files using `UTF-8` on windows.

    AWS_CLI_FILE_ENCODING=UTF-8

For more information, see [Environment variables to configure the AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-envvars.html) .

or other related to S3 provided in [Breaking changes -- Migrating from AWS CLI version 1 to version 2 docs](https://docs.aws.amazon.com/cli/latest/userguide/cliv2-migration.html)

Also, good to check object's key naming:
>
> When you create an object, you also specify the key name, which uniquely identifies the object in the bucket. The object key (or key name) uniquely identifies the object in an Amazon S3 bucket. For more information, see [Creating object key names](https://docs.aws.amazon.com/AmazonS3/latest/userguide/object-keys.html).

According to the logs provided we found multiple errors:

```
upload failed: <redacted file> 'utf-8' codec can't encode characters in position 92-94: surrogates not allowed
```

```
upload failed: <redacted file> 'utf-8' codec can't encode characters in position 12-13: surrogates not allowed
```

...

It would be nice if you:  
- check and provide \<redacted file\> names to found root case  
- try to provide AWS_CLI_FILE_ENCODING`=UTF-`8 as a pipe's variable

                - pipe: atlassian/aws-s3-deploy:1.0.0              variables:                S3_BUCKET: ${S3_BUCKET}/home                LOCAL_PATH: 'home'                ACL: 'public-read'                EXTRA_ARGS: '--exclude=*.html'
                    AWS_CLI_FILE_ENCODING: 'UTF-8'

Thank you for your contribution!

Best regards,  
Oleksandr Kyrdan


author: Leonardo Medici
body: [@Oleksandr Kyrdan](/t5/user/viewprofilepage/user-id/2885705)I tried to run the pipeline with AWS_CLI_FILE_ENCODING variable without success.

I don't know if object key is an issues, I tried to manually run S3 sync command with

```
aws-cli/2.2.15 Python/3.9.5 Darwin/20.5.0 source/x86_64 prompt/off
```

And at least on my platform this was not a problem.

I can try to remove any strange character from file names but this will take some time.


author: Oleksandr Kyrdan
body: Hi [@Leonardo Medici](/t5/user/viewprofilepage/user-id/4457604)

Some summary after the issue investigation,

<br />

Started from version 1.0.0 aws-s3-deploy pipe uses [AWS CLI version 2](https://docs.aws.amazon.com/cli/latest/userguide/welcome-versions.html).

So, now AWS CLI version 2 more strict for filenames/[key-names](https://docs.aws.amazon.com/AmazonS3/latest/userguide/object-keys.html): **only utf-8 supported.**

So, the official [AWS S3](https://docs.aws.amazon.com/cli/latest/userguide/cliv2-migration.html)[Creating object key names guide](https://docs.aws.amazon.com/cli/latest/userguide/cliv2-migration.html) should be updated by AWS to the next:
>
> Object key naming guidelines {#toc-hId--1240872429}
> ---------------------------------------------------
>
> You **can** **must** use any UTF-8 character in an object key name. However, using certain characters in key names can cause problems with some applications and protocols. The following guidelines help you maximize compliance with DNS, web-safe characters, XML parsers, and other APIs.

To solve your issue, try not to use non-**utf-8** character in an object key name (filename).

According to your question, we updated Readme for the pipe and released new version

    - pipe: atlassian/aws-s3-deploy:1.0.1

Thank you for your contribution!

Cheers,  
OIeksandr Kyrdan

comments: author: Leonardo Medici
body: [@Oleksandr Kyrdan](/t5/user/viewprofilepage/user-id/2885705)I was looking some minutes ago at your repository and found the note in the latest commits.

I also found out that some of our pdf files have non utf-8 charaters, probably the root cause was a copy-paste from MS-Word that use non standard charaters for ' " - (at least in my native language)

Changed the characters and now it's working like a charm.

Thank you for your help


Tags: aws

bitbucket-pipelines

bug

cloud