How to reuse some steps across pipelines but with minor changes
Asked by nagarro_bhavya:

I have a use case wherein my repository works on multiple AWS Lambda functions. I am using bitbucket pipelines to test each of those python functions (based on the changeset condition) with pytest and pylint and send an email in case any of those fail.

All the steps are exactly same for all the functions except some things such as

1. the path where to run pytest

2. the message to send in email etc.

My pipeline looks something like this:

```
- step:
? name: function 1
? caches:
    - pip
  script:
    - pip install --upgrade pip
    - other common steps
? ? - cd root_folder/function_1
? ? - pytest

    - other common steps
  condition:
    changesets:
      includePaths:
? ? ? ?- root_folder/function_1/**
? ? ? ? - utility/scripts/**
   after-script:
     - echo $BITBUCKET_EXIT_CODE

     - if [ $BITBUCKET_EXIT_CODE != 0 ]; then echo "Step failed"; 
     - pipe: atlassian/email-notify:0.4.4
     variables:
       USERNAME: some username
       PASSWORD: some password
       FROM: $FROM_EMAIL

       Other common variables
       SUBJECT: 'function 1 Failed'

       ATTACHMENTS:

       '/opt/atlassian/pipelines/agent/build/root_folder/function_1/coverage/index.html'
     - fi; 
   artifacts:
     - test-reports/* 


- step:
? name: function 2
? caches:
    - pip
  script:
    - pip install --upgrade pip
    - other common steps
? ? - cd root_folder/function_2
? ? - pytest

    - other common steps
  condition:
    changesets:
      includePaths:
? ? ? ?- root_folder/function_2/**
? ? ? ? - utility/scripts/**
   after-script:
     - echo $BITBUCKET_EXIT_CODE

     - if [ $BITBUCKET_EXIT_CODE != 0 ]; then echo "Step failed"; 
     - pipe: atlassian/email-notify:0.4.4
     variables:
       USERNAME: some username
       PASSWORD: some password
       FROM: $FROM_EMAIL

       Other common variables
       SUBJECT: 'function 2 Failed'

       ATTACHMENTS:

       '/opt/atlassian/pipelines/agent/build/root_folder/function_2/coverage/index.html'
     - fi; 
   artifacts:
     - test-reports/*

```

All the bold text is common to each step. Whereas only the italic part varies.

I have 10 such functions. I am looking a way out so that I dont have to repeat all these lines again and again. As you can see keeping two arguments which change with each step will solve the issue.

I cant use YAML anchors as there are a couple of changes here and there which are specific to each step.


Answers:
author: Theodora Boudale
body: Hi [@nagarro_bhavya](/t5/user/viewprofilepage/user-id/4483452) and welcome to the community.

I'm afraid that at the moment Bitbucket Pipelines doesn't support step variables. I believe that what you're asking is something similar to the following feature request?

* <https://jira.atlassian.com/browse/BCLOUD-20821>

The way to reuse code in Pipelines at the moment is with YAML anchors, and for your specific use case, my suggestion would be a combination of deployment environments+variables along with YAML anchors.

You mentioned that you cannot use YAML anchors as there are a couple of changes that are specific to each step. Could you please give an example of what kind of changes? Do you want to execute extra commands for certain steps? Or something different?

Kind regards,  
Theodora

comments: author: nagarro_bhavya
body: Hello [@Theodora Boudale](/t5/user/viewprofilepage/user-id/818086) ,

Thanks for your reply.

For example, in the above snippet, the underlined or non-bold text are the changes which are step-specific and hence using anchors will not be possible.


author: Theodora Boudale
body: Hi [@nagarro_bhavya](/t5/user/viewprofilepage/user-id/4483452),

Thanks for getting back to me.

It should be possible to use ++YAML anchors++ in combination with ++deployment environments++ and ++deployment variables++ to reuse code.

What ++cannot++ be included in the YAML anchor is the step name and the condition.

It is not possible to assign a variable to a step's name, and a variable used in a path of *includePaths:* won't work because the condition is used before the build execution, while the variables are part of a container during its execution only.

The rest of the content though can go in a yaml anchor.

Please allow me to give you an example with 2 functions, to show how you could achieve reuse of code:

**1.** In Bitbucket website, open the repo where you have this pipeline, go to its **Repository settings \> Deployments** and create ++one environment for each function++. In my case, I created 2, and I named them Function_1 and Function_2.

++For each one of these environments++ , create a variable with the name *function*, and assign it the respective value that corresponds to the name of the directory in your repo.

E.g. for the environment Function_1, I created a variable named *function* , with the value *function_1* .  
For the environment Function_2, I created a variable named *function* , with the value *function_2*.

It's important that this variable has the same name for all environments you'll use for your functions.

**2.** The YAML file can look as follows:

```
definitions: 
  steps:
    - step: &myfunction
        caches:
          - pip
        script:
          - pip install --upgrade pip
          - other common steps
          - cd root_folder/$function
          - pytest
          - other common steps
        after-script:
          - echo $BITBUCKET_EXIT_CODE
          - if [ $BITBUCKET_EXIT_CODE != 0 ]; then echo "Step failed"; 
          - pipe: atlassian/email-notify:0.4.4
            variables:
              USERNAME: some username
              PASSWORD: some password
              FROM: $FROM_EMAIL
              Other common variables
              SUBJECT: '${function} Failed'
              ATTACHMENTS: '/opt/atlassian/pipelines/agent/build/root_folder/${function}/coverage/index.html'
          - fi; 
        artifacts:
          - test-reports/*

pipelines:
  default:
    - step:
        <<: *myfunction
        condition:
          changesets:
            includePaths:
              - root_folder/function_1/**
              - utility/scripts/**
        name: function 1
        deployment: Function_1
    - step:
        <<: *myfunction
        condition:
          changesets:
            includePaths:
              - root_folder/function_2/**
              - utility/scripts/**
        name: function 2
        deployment: Function_2
```

The biggest part of your step, with the exception of the name and the condition, can go in a yaml anchor. Note the use of the variable **${function}** in the yaml anchor.

Later, when we reference the YAML anchor in a specific step, we also use the keyword **deployment** with a certain environment.

For the first step that has **deployment: Function_1** , the variable *${function}* in the script will take the value of the deployment variable *function* that we defined in Function_1 environment, which is function_1.

The second step has **deployment: Function_2** , so when the script runs, the variable *${function}* in the script will take the value of the deployment variable *function* that we defined in Function_2 environment, which is function_2.

We need to give the name of the step outside the anchor, as we can't assign variables to a step's name.  
Additionally, the condition: / changesets: / includePaths: must also be outside the anchor, as we cannot use a variable's name in the path. E.g. if we had something like - root_folder/$function/\*\* in a condition inside a step, it would not work.

So, even though certain things need to be repeated, you can reuse your script and after-script.

Is this something that would work for you?

Please feel free to let me know if you have any questions.

Kind regards,  
Theodora


author: nagarro_bhavya
body: Thank you [@Theodora Boudale](/t5/user/viewprofilepage/user-id/818086)

This answers my question,

Will I have to purchase the premium plan as I want to be able to run the deployment pipelines for any pull requests to a specific branch raised by anyone (non-admins as well).


author: Theodora Boudale
body: Hi [@nagarro_bhavya](/t5/user/viewprofilepage/user-id/4483452),

You are very welcome and I apologize for my late reply, it looks that I've been missing some notifications.

Regarding your last question:
> Will I have to purchase the premium plan as I want to be able to run the deployment pipelines for any pull requests to a specific branch raised by anyone (non-admins as well)

<br />

If you have the premium plan, you can restrict which branches the deployment can run for, e.g. the deployment can run only for master branch or development branch, if there are new commits to this branch. But then, the deployment needs to be specified for a branch in the yml file.

If you want to use the deployments in the 'pull-requests:' definition of the yml file, then you specify in the yml file the source branch (not the destination branch) and the builds run when you create the PR (not when it is merged to the destination branch). It is not possible to specify a restriction for pull-requests builds based on the destination branch, if this is your use case, so you wouldn't need the premium plan in this case.

Please feel free to let me know if you have any further questions.

Kind regards,  
Theodora


author: nagarro_bhavya
body: Hey [@Theodora Boudale](/t5/user/viewprofilepage/user-id/818086)

I just read 12 days of CI/CD series and came across the concept of Default Variables as mentioned here: <https://bitbucket.org/blog/updates-to-bitbucket-pipes>

Can you please provide an example of how can I use it. As you see, in my case, I have to use a pipe multiple times which has 3-4 variables in common. Hence, it will help me a lot.


Tags: bitbucket-pipelines

bitbucket-cloud

cloud

continuous-delivery