Pipeline error: yaml.scanner.ScannerError: mapping values are not allowed
Asked by Srinivas Nekkalapudi:

Sometimes pipeline is successful and a few times it is failing.

I assume the dynamic BITBUCKET_BRANCH name might be the issue.

    INFO: Successfully updated the kube config.

    Traceback (most recent call last):

      File "/pipe.py", line 47, in <module>

        pipe.run()

      File "/root/.local/lib/python3.8/site-packages/kubectl_run/pipe.py", line 157, in run

        self.handle_apply()

      File "/root/.local/lib/python3.8/site-packages/kubectl_run/pipe.py", line 114, in handle_apply

        self.update_labels_in_metadata(template_file, labels)

      File "/root/.local/lib/python3.8/site-packages/kubectl_run/pipe.py", line 30, in update_labels_in_metadata

        yamls = list(yaml.safe_load_all(template_file.read()))

      File "/root/.local/lib/python3.8/site-packages/yaml/__init__.py", line 130, in load_all

        yield loader.get_data()

      File "/root/.local/lib/python3.8/site-packages/yaml/constructor.py", line 45, in get_data

        return self.construct_document(self.get_node())

      File "/root/.local/lib/python3.8/site-packages/yaml/composer.py", line 27, in get_node

        return self.compose_document()

      File "/root/.local/lib/python3.8/site-packages/yaml/composer.py", line 55, in compose_document

        node = self.compose_node(None, None)

      File "/root/.local/lib/python3.8/site-packages/yaml/composer.py", line 84, in compose_node

        node = self.compose_mapping_node(anchor)

      File "/root/.local/lib/python3.8/site-packages/yaml/composer.py", line 133, in compose_mapping_node

        item_value = self.compose_node(node, item_key)

      File "/root/.local/lib/python3.8/site-packages/yaml/composer.py", line 84, in compose_node

        node = self.compose_mapping_node(anchor)

      File "/root/.local/lib/python3.8/site-packages/yaml/composer.py", line 133, in compose_mapping_node

        item_value = self.compose_node(node, item_key)

      File "/root/.local/lib/python3.8/site-packages/yaml/composer.py", line 84, in compose_node

        node = self.compose_mapping_node(anchor)

      File "/root/.local/lib/python3.8/site-packages/yaml/composer.py", line 133, in compose_mapping_node

        item_value = self.compose_node(node, item_key)

      File "/root/.local/lib/python3.8/site-packages/yaml/composer.py", line 84, in compose_node

        node = self.compose_mapping_node(anchor)

      File "/root/.local/lib/python3.8/site-packages/yaml/composer.py", line 133, in compose_mapping_node

        item_value = self.compose_node(node, item_key)

      File "/root/.local/lib/python3.8/site-packages/yaml/composer.py", line 82, in compose_node

        node = self.compose_sequence_node(anchor)

      File "/root/.local/lib/python3.8/site-packages/yaml/composer.py", line 111, in compose_sequence_node

        node.value.append(self.compose_node(node, index))

      File "/root/.local/lib/python3.8/site-packages/yaml/composer.py", line 84, in compose_node

        node = self.compose_mapping_node(anchor)

      File "/root/.local/lib/python3.8/site-packages/yaml/composer.py", line 127, in compose_mapping_node

        while not self.check_event(MappingEndEvent):

      File "/root/.local/lib/python3.8/site-packages/yaml/parser.py", line 98, in check_event

        self.current_event = self.state()

      File "/root/.local/lib/python3.8/site-packages/yaml/parser.py", line 428, in parse_block_mapping_key

        if self.check_token(KeyToken):

      File "/root/.local/lib/python3.8/site-packages/yaml/scanner.py", line 116, in check_token

        self.fetch_more_tokens()

      File "/root/.local/lib/python3.8/site-packages/yaml/scanner.py", line 223, in fetch_more_tokens

        return self.fetch_value()

      File "/root/.local/lib/python3.8/site-packages/yaml/scanner.py", line 577, in fetch_value

        raise ScannerError(None, None,

    yaml.scanner.ScannerError: mapping values are not allowed here

      in "<unicode string>", line 30, column 79:

         ... -1.amazonaws.com/aupris-frontend:

bitbucker-pipeline file

```
image: atlassian/default-image:2

options:     

  docker: true

  size: 2x

pipelines:

  tags:

    release-*:

      - step:

          name: Build docker image

          services:

            - docker

          script:

          # build the image

          - docker build -t aupris-frontend .

          # use the pipe to push the image to AWS ECR

          - pipe: atlassian/aws-ecr-push-image:1.4.2

            variables:

              AWS_ACCESS_KEY_ID: $AWS_ACCESS_KEY_ID

              AWS_SECRET_ACCESS_KEY: $AWS_SECRET_ACCESS_KEY

              AWS_DEFAULT_REGION: $AWS_DEFAULT_REGION

              IMAGE_NAME: aupris-frontend

              TAGS: '${BITBUCKET_BRANCH}'

          - apt-get update && apt-get install gettext-base

          - envsubst < aupris-frontend.tpl.yaml > aupris-frontend.yaml

          - sed -ie "s/THIS_STRING_IS_REPLACED_DURING_BUILD/$(date)/g" aupris-frontend.yaml

          - pipe: "atlassian/aws-eks-kubectl-run:2.2.0"

            variables:

              AWS_ACCESS_KEY_ID: $AWS_ACCESS_KEY_ID

              AWS_SECRET_ACCESS_KEY: $AWS_SECRET_ACCESS_KEY

              AWS_DEFAULT_REGION: XXXXX

              CLUSTER_NAME: XXXXX

              KUBECTL_COMMAND: "apply"

              RESOURCE_PATH: "aupris-frontend.yaml"

              DEBUG: "true"

definitions:

  services:

    docker:

      memory: 7128
```

yaml file  

```
apiVersion: v1

kind: Service

metadata:

  name: aupris-frontend

spec:

  # type: LoadBalancer

  selector:

    app: aupris-frontend

  ports:

    - name: http

      port: 80

      targetPort: 80

---

apiVersion: apps/v1

kind: Deployment

metadata:

  name: aupris-frontend

spec:

  selector:

    matchLabels:

      app: aupris-frontend

  template:

    metadata:

      labels:

        app: aupris-frontend

    spec:

      terminationGracePeriodSeconds: 30

      containers:

        - name: aupris-frontend

          image: XXXXXXXXX:${BITBUCKET_BRANCH}

          imagePullPolicy: "Always"

          env:

            - name: API_URL

              value: api/

            - name: FOR_GODS_SAKE_PLEASE_REDEPLOY

              value: "THIS_STRING_IS_REPLACED_DURING_BUILD"

          resources:

            limits:

              memory: "128Mi"

              cpu: "100m"

          ports:

            - containerPort: 80
```


Answers:
author: Igor Stoyanov
body: [@Srinivas Nekkalapudi](/t5/user/viewprofilepage/user-id/2462204) hi.  

You should not use dynamical variables inside your aupris-frontend yaml Kubernetes manifest file. And As [@Oday Rafeh](/t5/user/viewprofilepage/user-id/5172837) mentioned, you could create template instead, fill it with needed variables and create according manifest. Also, optionally, you could validate your generated manifest.  

Regards, Igor


author: Erez Maadani
body: Hi [@Srinivas Nekkalapudi](/t5/user/viewprofilepage/user-id/2462204) ,

1. It seems you have different python versions. Your original post shows python3.8 while the screenshot of the "pip install.." shows python2.7. You might need a different runner image for your pipeline.
2. 2+3 - Every step of the pipeline is running in different container, so by default, env parameters and files are not shared. You might need to define a cache or artifacts to pass files between steps. Once you can share files, you can also share env parameters using files, just dump and read to/from a shard file. This link might help: <https://community.atlassian.com/t5/Bitbucket-questions/caches-vs-artifacts/qaq-p/864904>


author: Oday Rafeh
body: [@Srinivas Nekkalapudi](/t5/user/viewprofilepage/user-id/2462204)

The error is due to a problem in your YAML file. To fix it:

Check for missing colons.  
Ensure proper indentation.  
Quote strings with special characters.  
Fix unintended line breaks.  
Review the YAML file, fix the issues, and retry running the pipeline.

IF this didn't help please provide the YAML file content so I can help more.

comments: author: Srinivas Nekkalapudi
body: [@Oday Rafeh](/t5/user/viewprofilepage/user-id/5172837) Thanks for the quick response. I have edited my post and updated yaml file.

I wonder why my pipeline is successful few time with the same configuration and failing few other times.

**I could not reply to your answer so editing my old comment.**

I came across another error with your solution:

    + yamllint aupris-server.yaml

    Traceback (most recent call last):

      File "/usr/bin/yamllint", line 5, in <module>

        from pkg_resources import load_entry_point

    ImportError: No module named 'pkg_resources'

![image.png](https://community.atlassian.com/t5/image/serverpage/image-id/246705i63B3D094BA0D0185/image-size/large?v=v2&px=999 "image.png")

---

[@Oday Rafeh](/t5/user/viewprofilepage/user-id/5172837) It still did not solve the issue. following is the screenshot:

![image.png](https://community.atlassian.com/t5/image/serverpage/image-id/246715iE46E9D1DD840D013/image-size/large?v=v2&px=999 "image.png")


author: Oday Rafeh
body: I noticed you have the image URL redacted with XXXXXXXXX. If the actual image URL in your YAML file contains any special characters, like colons, you need to wrap the value in quotes.

The updated section with quotes:

```
 containers:
- name: aupris-frontend
image: "XXXXXXXXX:${BITBUCKET_BRANCH}"
imagePullPolicy: "Always"
```


author: Oday Rafeh
body: To address the intermittent pipeline failures related to the dynamic BITBUCKET_BRANCH name, you can modify your bitbucket-pipelines. yml file by adding a step to log the branch name, validate the YAML file using yamllint, and replace special characters in the branch name with underscores.

Updated bitbucket-pipelines. yml:

```
image: atlassian/default-image:2

options:
docker: true
size: 2x

pipelines:
tags:
release-*:
- step:
name: Prepare and validate
script:
- echo "BITBUCKET_BRANCH: ${BITBUCKET_BRANCH}"
- apt-get update && apt-get install -y gettext-base yamllint
- export BITBUCKET_BRANCH_SAFE="$(echo ${BITBUCKET_BRANCH} | sed 's/[^a-zA-Z0-9]/_/g')"
- echo "BITBUCKET_BRANCH_SAFE: ${BITBUCKET_BRANCH_SAFE}"
- envsubst < aupris-frontend.tpl.yaml > aupris-frontend.yaml
- yamllint aupris-frontend.yaml
- step:
name: Build docker image
services:
- docker
script:
# build the image
- docker build -t aupris-frontend .
# use the pipe to push the image to AWS ECR
- pipe: atlassian/aws-ecr-push-image:1.4.2
variables:
AWS_ACCESS_KEY_ID: $AWS_ACCESS_KEY_ID
AWS_SECRET_ACCESS_KEY: $AWS_SECRET_ACCESS_KEY
AWS_DEFAULT_REGION: $AWS_DEFAULT_REGION
IMAGE_NAME: aupris-frontend
TAGS: '${BITBUCKET_BRANCH_SAFE}'
- sed -ie "s/THIS_STRING_IS_REPLACED_DURING_BUILD/$(date)/g" aupris-frontend.yaml
- pipe: "atlassian/aws-eks-kubectl-run:2.2.0"
variables:
AWS_ACCESS_KEY_ID: $AWS_ACCESS_KEY_ID
AWS_SECRET_ACCESS_KEY: $AWS_SECRET_ACCESS_KEY
AWS_DEFAULT_REGION: XXXXX
CLUSTER_NAME: XXXXX
KUBECTL_COMMAND: "apply"
RESOURCE_PATH: "aupris-frontend.yaml"
DEBUG: "true"

definitions:
services:
docker:
memory: 7128
```

I added new step called Prepare and validate". This step logs the branch name, installs yamllint, and creates a sanitized version of the branch name called BITBUCKET_BRANCH_SAFE. The branch name is then used in the subsequent steps for building the Docker image and deploying it.

Please update the aupris-frontend. tpl. yaml file to use ${BITBUCKET_BRANCH_SAFE} instead of ${BITBUCKET_BRANCH} for the image name:

```
# ... rest of the file
containers:
- name: aupris-frontend
image: "XXXXXXXXX:${BITBUCKET_BRANCH_SAFE}"
imagePullPolicy: "Always"
# ... rest of the file
```


author: Oday Rafeh
body: Hi [@Srinivas Nekkalapudi](/t5/user/viewprofilepage/user-id/2462204) , It seems that the 'pkg_resources' module is missing from your environment. You can install it by adding this command to your pipeline script:

```
pip install setuptools
```

The fully updated "Prepare and validate" step should be like this :

```
- step:
name: Prepare and validate
script:
- echo "BITBUCKET_BRANCH: ${BITBUCKET_BRANCH}"
- apt-get update && apt-get install -y gettext-base yamllint
- pip install setuptools
- export BITBUCKET_BRANCH_SAFE="$(echo ${BITBUCKET_BRANCH} | sed 's/[^a-zA-Z0-9]/_/g')"
- echo "BITBUCKET_BRANCH_SAFE: ${BITBUCKET_BRANCH_SAFE}"
- envsubst < aupris-frontend.tpl.yaml > aupris-frontend.yaml
- yamllint aupris-frontend.yaml
```


author: Srinivas Nekkalapudi
body: [@Oday Rafeh](/t5/user/viewprofilepage/user-id/5172837) Please have a look at the screenshot.

![image.png](https://community.atlassian.com/t5/image/serverpage/image-id/246723iBD03A8E453DFF0A3/image-size/large?v=v2&px=999 "image.png")


author: Oday Rafeh
body: [@Srinivas Nekkalapudi](/t5/user/viewprofilepage/user-id/2462204)

Try this please should work based on your screenshot :

```
sudo apt-get install python-setuptools
```

Prepare and validate" step should be like this now :

```
- step:
name: Install dependencies
script:
- apt-get update && apt-get install -y python-setuptools
```


author: Srinivas Nekkalapudi
body: [@Oday Rafeh](/t5/user/viewprofilepage/user-id/5172837) Even with that it did not work

![image.png](https://community.atlassian.com/t5/image/serverpage/image-id/246727i107B6AD28D6BEB6F/image-size/large?v=v2&px=999 "image.png")

Found one more issue with the pipeline code:

the file could not be found.

BTW I skipped yamllint line to go further into other steps.

![image.png](https://community.atlassian.com/t5/image/serverpage/image-id/246728i5A4DA17A1CCF52C8/image-size/large?v=v2&px=999 "image.png")

here is the bitbucket pipeline code:  

```
image: atlassian/default-image:2

pipelines:

  tags:

    release-*:

      - step:

          name: Prepare and validate

          script:

            - echo "BITBUCKET_BRANCH  ${BITBUCKET_BRANCH}"

            - apt-get update && apt-get install -y python-setuptools gettext-base yamllint

            - export BITBUCKET_BRANCH_SAFE="$(echo ${BITBUCKET_BRANCH} | sed 's/[^a-zA-Z0-9]/_/g')"

            - echo "BITBUCKET_BRANCH_SAFE ${BITBUCKET_BRANCH_SAFE}"

            - envsubst < aupris-server.tpl.yaml > aupris-server.yaml

            - yamllint aupris-server.yaml

      - step:

          name: Build docker image and push to ECR

          services:

            - docker

          script:

          # build the image

          - docker build -t aupris-server .

          # use the pipe to push the image to AWS ECR

          - pipe: atlassian/aws-ecr-push-image:1.4.2

            variables:

              AWS_ACCESS_KEY_ID: $AWS_ACCESS_KEY_ID

              AWS_SECRET_ACCESS_KEY: $AWS_SECRET_ACCESS_KEY

              AWS_DEFAULT_REGION: $AWS_DEFAULT_REGION

              IMAGE_NAME: aupris-server

              TAGS: '${BITBUCKET_BRANCH_SAFE}'

      - step:

          name: Releasing to kubernetes via EKS

          script: 

          - sed -ie "s/THIS_STRING_IS_REPLACED_DURING_BUILD/$(date)/g" aupris-server.yaml

          - pipe: "atlassian/aws-eks-kubectl-run:2.2.0"

            variables:

              AWS_ACCESS_KEY_ID: $AWS_ACCESS_KEY_ID

              AWS_SECRET_ACCESS_KEY: $AWS_SECRET_ACCESS_KEY

              AWS_DEFAULT_REGION: $AWS_DEFAULT_REGION

              CLUSTER_NAME: "julius66"

              KUBECTL_COMMAND: "apply"

              RESOURCE_PATH: "aupris-server.yaml"

              DEBUG: "true"
```


author: Oday Rafeh
body: [@Srinivas Nekkalapudi](/t5/user/viewprofilepage/user-id/2462204)

The error "sed: can't read aupris-server. yaml: No such file or directory" indicates that the file 'aupris-server. yaml' is not found in the current directory.

You should ensure that the 'aupris-server. tpl. yaml' file is present in the same directory as the bitbucket-pipelines. yml file, and that it has been correctly defined to generate the 'aupris-server. yaml' file.

You can also check the output of the 'Prepare and validate' step to see if there were any errors related to the creation of the aupris-server. yaml file.


author: Srinivas Nekkalapudi
body: [@Oday Rafeh](/t5/user/viewprofilepage/user-id/5172837)

aupris-server.yaml and bitbucket-pipelines. yml are in the same directory.

there are no errors in 'Prepare and validate' step other than

    ImportError: No module named 'pkg_resources'

Is it because of the STEPS?

when i do echo "BITBUCKET_BRANCH_SAFE ${BITBUCKET_BRANCH_SAFE}" in step "Releasing to Kubernetes via EKS" it gives me an output like "BITBUCKET_BRANCH_SAFE ".


author: Oday Rafeh
body: [@Srinivas Nekkalapudi](/t5/user/viewprofilepage/user-id/2462204)

The "BITBUCKET_BRANCH_SAFE" variable seems to be empty in the "Releasing to Kubernetes via EKS" step, which could be causing the issue with the sed command. One possible reason for this could be that the environment variable is not being passed to the subsequent step properly.

To troubleshoot this issue, you can try adding a debug statement in the "Build docker image and push to ECR" step to confirm that the "BITBUCKET_BRANCH_SAFE" variable is being properly set, If you want help with that let me know I am here to help


author: Srinivas Nekkalapudi
body: [@Oday Rafeh](/t5/user/viewprofilepage/user-id/5172837) It's empty in step "Build docker image and push to ECR" as well

![image.png](https://community.atlassian.com/t5/image/serverpage/image-id/246742iCE956FC77CEA0BFE/image-size/large?v=v2&px=999 "image.png")


author: Oday Rafeh
body: [@Srinivas Nekkalapudi](/t5/user/viewprofilepage/user-id/2462204)

the issue is with the "Build docker image and push to ECR" step. The docker build command seems to be missing the context path and is not able to find the Dockerfile.

Please update the pipeline code

```
image: atlassian/default-image:2

pipelines:

tags:

release-*:

- step:

name: Prepare and validate

script:

- echo "BITBUCKET_BRANCH ${BITBUCKET_BRANCH}"

- apt-get update && apt-get install -y python-setuptools gettext-base yamllint

- export BITBUCKET_BRANCH_SAFE="$(echo ${BITBUCKET_BRANCH} | sed 's/[^a-zA-Z0-9]/_/g')"

- echo "BITBUCKET_BRANCH_SAFE ${BITBUCKET_BRANCH_SAFE}"

- envsubst < aupris-server.tpl.yaml > aupris-server.yaml

- yamllint aupris-server.yaml

- step:

name: Build docker image and push to ECR

services:

- docker

script:

# build the image

- docker build -t aupris-server -f Dockerfile .

# use the pipe to push the image to AWS ECR

- pipe: atlassian/aws-ecr-push-image:1.4.2

variables:

AWS_ACCESS_KEY_ID: $AWS_ACCESS_KEY_ID

AWS_SECRET_ACCESS_KEY: $AWS_SECRET_ACCESS_KEY

AWS_DEFAULT_REGION: $AWS_DEFAULT_REGION

IMAGE_NAME: aupris-server

TAGS: '${BITBUCKET_BRANCH_SAFE}'

- step:

name: Releasing to kubernetes via EKS

script:

- sed -ie "s/THIS_STRING_IS_REPLACED_DURING_BUILD/$(date)/g" aupris-server.yaml

- pipe: "atlassian/aws-eks-kubectl-run:2.2.0"

variables:

AWS_ACCESS_KEY_ID: $AWS_ACCESS_KEY_ID

AWS_SECRET_ACCESS_KEY: $AWS_SECRET_ACCESS_KEY

AWS_DEFAULT_REGION: $AWS_DEFAULT_REGION

CLUSTER_NAME: "julius66"

KUBECTL_COMMAND: "apply"

RESOURCE_PATH: "aupris-server.yaml"

DEBUG: "true"
```

I added the "-f Dockerfile" option to the docker build command to specify the location of the Dockerfile, which should be in the same directory as the pipeline file. This should resolve the issue with the empty output


author: Srinivas Nekkalapudi
body: [@Oday Rafeh](/t5/user/viewprofilepage/user-id/5172837) I don't think there is anything wrong with the context path.

![image.png](https://community.atlassian.com/t5/image/serverpage/image-id/246748i385401E9F92ECA0E/image-size/large?v=v2&px=999 "image.png")


author: Oday Rafeh
body: [@Srinivas Nekkalapudi](/t5/user/viewprofilepage/user-id/2462204)

Correct, The output from the pipeline suggests that the Docker image is being built successfully and the BITBUCKET_BRANCH_SAFE variable is being printed correctly. The pipeline then goes on to push the Docker image to AWS ECR using the atlassian/aws-ecr-push-image pipe. Without more information, it's difficult to determine if there are any issues with this step.


author: Oday Rafeh
body: [@Srinivas Nekkalapudi](/t5/user/viewprofilepage/user-id/2462204) ,

I was thinking a little bit out of the box and I tried to validate your YAML code

here its :

![2023-03-21 13_13_49-Best YAML Validator Online.png](https://community.atlassian.com/t5/image/serverpage/image-id/246749i91796B2284D1C321/image-size/large?v=v2&px=999 "2023-03-21 13_13_49-Best YAML Validator Online.png") The error seems to be related to the YAML syntax of the aupris-frontend. yaml file. The error message specifically mentions that "mapping values are not allowed here". This error usually occurs when there is a syntax error in the YAML file.

This error typically occurs when there are multiple YAML documents in a single file. Each YAML file should contain only one document. Please check your YAML file and make sure that it contains only one document. If you have multiple documents, you can separate them by adding three hyphens (---) between each document.


author: Oday Rafeh
body: So lets Try this file instead :

```
---
apiVersion: v1
kind: Service
metadata:
name: aupris-frontend
spec:
selector:
app: aupris-frontend
ports:
- name: http
port: 80
targetPort: 80

---
apiVersion: apps/v1
kind: Deployment
metadata:
name: aupris-frontend
spec:
selector:
matchLabels:
app: aupris-frontend
template:
metadata:
labels:
app: aupris-frontend
spec:
terminationGracePeriodSeconds: 30
containers:
- name: aupris-frontend
image: XXXXXXXXX:${BITBUCKET_BRANCH}
imagePullPolicy: Always
env:
- name: API_URL
value: api/
- name: FOR_GODS_SAKE_PLEASE_REDEPLOY
value: THIS_STRING_IS_REPLACED_DURING_BUILD
resources:
limits:
memory: 128Mi
cpu: 100m
ports:
- containerPort: 80
```


author: Srinivas Nekkalapudi
body: [@Oday Rafeh](/t5/user/viewprofilepage/user-id/5172837) HI, If you see my original post I have a single file with two documents separated by (---).

so there are no changes to make I believe.

To summarise, the following are the issues I have:

* ImportError: No module named 'pkg_resources'
* Could not access BITBUCKET_BRANCH_SAFE in steps 2 and 3.
* Newly created files(e.g: sed -ie) could not be accessed in steps 2 and 3.


Tags: atl-gruland

bitbucket-cloud

cloud