Fetch selective secrets with no access to key vault using Devops pipelines
Tags: azure, devops, powershell, iac
Published: 2022-04-21 12:10:00

Greetings my fellow Technology Advocates and Specialists!!!

This Blog post is a follow-up to **<u>my previous post</u>** - [Fetch Secrets with no access to key vault] (https://dev.to/arindam0310018/fetch-secrets-with-no-access-to-key-vault-using-devops-pipelines-54h3)

| **MY SUGGESTION:-** | 
| --------- |
| Please Read my previous blog first to better understand the Use Case |

| **USE CASES:-** | 
| --------- |
| Fetch Secrets only with Tags (any) |
| Fetch Secrets only with Specific defined Tags |

| __LIVE RECORDED SESSION:-__ |
| --------- |
| __LIVE DEMO__ was Recorded as part of my Presentation in __WELSH AZURE GROUP__ Forum/Platform |
| Duration of My Demo = __32 Mins 38 Secs__ |
| Start and End Time = __00:39:22 to 01:12:00__ |

{% youtube um_6WtIBSA8 %}

| __REQUIREMENTS:-__ |
| --------- |

1. Azure Key Vault
2. Four Sample Secrets in Azure Key Vault
3. Add Tags to One or More Secrets 
4. Azure Resource Manager Service Connection 
5. Azure DevOps Pipeline (YAML)

| **NOTE:-** |
| --------- |

The Service Principal (which is required to Create Service Connection) should at minimum have **GET** and **LIST** Access Policy Permissions in Azure Key Vault.

| __BELOW DISPLAYS THE SAMPLE SECRETS IN KEY VAULT:-__ |
| --------- |
| ![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/npsfbj72ds452270pbmk.png)  |

| __WHERE DO WE APPLY TAGS IN KEY VAULT SECRET:-__ |
| --------- |
| ![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/myl91y0jliypmxvjx6dt.png) |
| ![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/kendex7v7zg3e6kk8vqy.png) |

| __BELOW DISPLAYS THE SECRET TAGS IN KEY VAULT:-__ |
| --------- |

| NAME OF THE SECRET | TAGS ASSOCIATED WITH SECRET |
| --------- | --------- |
| AMSecret005 | ![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/0fjqve09z2qxhb2d96ei.png) |
| AMSecret006 | ![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/cq2mqytunhw7bl3zxwwe.png) |
| AMSecret007 | No Tags Configured |
| AMSecret008 | ![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/0fjqve09z2qxhb2d96ei.png) |

| # | PIPELINE TASKS | 
| --------- | --------- |
| 1. | AZURE KEY VAULT TASKS |
| 2. | FETCH **ALL SECRETS WITH TAGS** OR **ALL SECRETS WITH SPECIFIC TAG**. EXPORT IT THEN IN A TEXT FILE | 
| 3. | COPY THE SECRETS TEXT FILE TO ARTIFACTS STAGING DIRECTORY |
| 4. | PUBLISH THE ARTIFACTS |

| __CODE REPOSITORY:-__ |
| --------- |
| {% github arindam0310018/21-Apr-2022-DevOps__Fetch-Selective-Secrets-With-No-Access-To-KeyVault
 %} |

| __Below follows the contents of the YAML File (Azure DevOps):-__ |
| --------- |

```
trigger:
  none

######################
#DECLARE VARIABLES:-
######################
variables:
  SubscriptionID: 210e66cb-55cf-424e-8daa-6cad804ab604
  ServiceConnection: amcloud-cicd-service-connection
  KVName: ampockv
  Artifact: AM
  KVTag: "RequiredBy: CloudProvider" 
  KVNoTag: "{}"
  
#########################
# Declare Build Agents:-
#########################
pool:
  vmImage: windows-latest

###################
# Declare Stages:-
###################
stages:

- stage: USECASE_DISPLAY_ALL_SECRETS_AND_VALUES
  jobs:
  - job: DISPLAY_SECRETS_AND_VALUES
    displayName: DISPLAY SECRETS AND VALUES
    steps:

########################################################################
# Azure Key Vault Task.
# Display the name of Key Vault.
# Display the No. of Secrets found in Key Vault.
# Display the No. of enabled and unexpired Secrets found in Key Vault.
# Downloads values of Each Secret in Key Vault.
########################################################################

    - task: AzureKeyVault@2
      displayName: AZ KEYVAULT TASK
      inputs:
        azureSubscription: '$(ServiceConnection)'
        KeyVaultName: '$(KVName)'
        SecretsFilter: '*'
        RunAsPreJob: false

#######################################################
# Integers can be compared with these operators:
#   -eq # Equal
#   -ne # Not equal
#   -lt # Less than
#   -le # Less than or equal
#   -gt # Greater than
#   -ge # Greater than or equal
#######################################################

###############################################################
# Export the Secret with Tags only to Secrets Text File
# Copy the Secrets Text file to Artifacts Staging Directory.
###############################################################

    # - task: AzureCLI@2
    #   displayName: FETCH ALL SECRETS WITH TAGS
    #   inputs:
    #     azureSubscription: '$(ServiceConnection)'
    #     scriptType: ps
    #     scriptLocation: inlineScript
    #     inlineScript: |
    #       az --version
    #       az account set --subscription $(SubscriptionID)
    #       az account show
    #       $count = az keyvault secret list --vault-name $(KVName) --query "[] | length(@)"
    #       For ($i=0; $i -lt $count; $i++) {
    #       $sectag = az keyvault secret list --vault-name $(KVName) --query [$i].tags -o yaml
    #       if($sectag -eq "$(KVNoTag)") {
    #       $secretname = az keyvault secret list --vault-name $(KVName) --query [$i].name -o tsv
    #       echo "THERE IS NO TAGS FOR THE $secretname"
    #         }
    #       else {
    #       $secretname = az keyvault secret list --vault-name $(KVName) --query [$i].name -o tsv
    #       $secretvalue = az keyvault secret show --vault-name $(KVName) --name $secretname --query value -o tsv
    #       echo ($secretname + ':' + $secretvalue) >> Secrets.txt
    #       echo "VALUE of $secretname IS EXPORTED TO SECRETS TEXT FILE"
    #       }
    #       }

#################################################################################################
# Export the Secret with Specific Tag "RequiredBy: CloudProvider" only to Secrets Text File
# Copy the Secrets Text file to Artifacts Staging Directory.
#################################################################################################

    - task: AzureCLI@2
      displayName: FETCH ALL SECRETS WITH SPECIFIC TAGS
      inputs:
        azureSubscription: '$(ServiceConnection)'
        scriptType: ps
        scriptLocation: inlineScript
        inlineScript: |
          az --version
          az account set --subscription $(SubscriptionID)
          az account show
          $count = az keyvault secret list --vault-name $(KVName) --query "[] | length(@)"
          For ($i=0; $i -lt $count; $i++) {
          $sectag = az keyvault secret list --vault-name $(KVName) --query [$i].tags -o yaml
          if($sectag -eq "$(KVNoTag)") {
          $secretname = az keyvault secret list --vault-name $(KVName) --query [$i].name -o tsv
          echo "THERE IS NO TAGS FOR THE $secretname"
            }
          elseif ($sectag -eq "$(KVTag)") {
          $secretname = az keyvault secret list --vault-name $(KVName) --query [$i].name -o tsv
          $secretvalue = az keyvault secret show --vault-name $(KVName) --name $secretname --query value -o tsv
          echo ($secretname + ':' + $secretvalue) >> Secrets.txt
          echo "VALUE of $secretname IS EXPORTED TO SECRETS TEXT FILE"
            }
          else {
          $secretname = az keyvault secret list --vault-name $(KVName) --query [$i].name -o tsv
          echo "There are other TAG(S) configured apart from Tag "$(KVTag)" for $secretname"
          }
          }  

###############################################################
# Copy the Secrets text file to Artifacts Staging Directory:-
###############################################################

    - task: CopyFiles@2
      displayName: COPY TO ARTIFACTS STAGING DIRECTORY
      inputs:
        Contents: Secrets.txt
        targetFolder: '$(Build.ArtifactStagingDirectory)'

###########################
# Publish the Artifacts:-
###########################

    - task: PublishBuildArtifacts@1
      displayName: PUBLISH ARTIFACTS
      inputs:
        PathtoPublish: '$(Build.ArtifactStagingDirectory)'
        ArtifactName: '$(Artifact)'
        publishLocation: 'Container'
```
| __Values of the VARIABLES incase if you wish to change:-__ |
| --------- |

```
######################
#DECLARE VARIABLES:-
######################
variables:
  SubscriptionID: 210e66cb-55cf-424e-8daa-6cad804ab604
  ServiceConnection: amcloud-cicd-service-connection
  KVName: ampockv
  Artifact: AM
  KVTag: "RequiredBy: CloudProvider" 
  KVNoTag: "{}"

```

| __Purpose of the VARIABLE "KVNoTag" :-__ |
| --------- |
| If the Key Vault Secret Version does not have any Tag configured, it still returns value = { }. Hence we defined it as a  variable and during runtime, we compare the output to the value of the variable. If the Value matches, we then know that the Secret Version does not have any Tag Configured.|
| ![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/v0g4pc5iacrfa5tc08kq.png) |
| ![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/g45on6o2wsqkeljcpicy.png) |

| __Change the Agent Pool from MICROSOFT HOSTED BUILD AGENT to SELF HOSTED BUILD AGENT otherwise, below error is encountered:-__ |
| --------- |

| ![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/jnakxms8lvplnx62cd8w.png) |
| --------- |

| __Below follows the code snippet for Use Case #1 (Fetch Secrets only with Tags [any]):-__ |
| --------- |

```
###############################################################
# Export the Secret with Tags only to Secrets Text File
# Copy the Secrets Text file to Artifacts Staging Directory.
###############################################################

    # - task: AzureCLI@2
    #   displayName: FETCH ALL SECRETS WITH TAGS
    #   inputs:
    #     azureSubscription: '$(ServiceConnection)'
    #     scriptType: ps
    #     scriptLocation: inlineScript
    #     inlineScript: |
    #       az --version
    #       az account set --subscription $(SubscriptionID)
    #       az account show
    #       $count = az keyvault secret list --vault-name $(KVName) --query "[] | length(@)"
    #       For ($i=0; $i -lt $count; $i++) {
    #       $sectag = az keyvault secret list --vault-name $(KVName) --query [$i].tags -o yaml
    #       if($sectag -eq "$(KVNoTag)") {
    #       $secretname = az keyvault secret list --vault-name $(KVName) --query [$i].name -o tsv
    #       echo "THERE IS NO TAGS FOR THE $secretname"
    #         }
    #       else {
    #       $secretname = az keyvault secret list --vault-name $(KVName) --query [$i].name -o tsv
    #       $secretvalue = az keyvault secret show --vault-name $(KVName) --name $secretname --query value -o tsv
    #       echo ($secretname + ':' + $secretvalue) >> Secrets.txt
    #       echo "VALUE of $secretname IS EXPORTED TO SECRETS TEXT FILE"
    #       }
    #       }

```
| __Below follows the code snippet for Use Case #2 (Fetch Secrets only with Specific defined Tags [RequiredBy: CloudProvider]):-__ |
| --------- |

```
#################################################################################################
# Export the Secret with Specific Tag "RequiredBy: CloudProvider" only to Secrets Text File
# Copy the Secrets Text file to Artifacts Staging Directory.
#################################################################################################

    - task: AzureCLI@2
      displayName: FETCH ALL SECRETS WITH SPECIFIC TAGS
      inputs:
        azureSubscription: '$(ServiceConnection)'
        scriptType: ps
        scriptLocation: inlineScript
        inlineScript: |
          az --version
          az account set --subscription $(SubscriptionID)
          az account show
          $count = az keyvault secret list --vault-name $(KVName) --query "[] | length(@)"
          For ($i=0; $i -lt $count; $i++) {
          $sectag = az keyvault secret list --vault-name $(KVName) --query [$i].tags -o yaml
          if($sectag -eq "$(KVNoTag)") {
          $secretname = az keyvault secret list --vault-name $(KVName) --query [$i].name -o tsv
          echo "THERE IS NO TAGS FOR THE $secretname"
            }
          elseif ($sectag -eq "$(KVTag)") {
          $secretname = az keyvault secret list --vault-name $(KVName) --query [$i].name -o tsv
          $secretvalue = az keyvault secret show --vault-name $(KVName) --name $secretname --query value -o tsv
          echo ($secretname + ':' + $secretvalue) >> Secrets.txt
          echo "VALUE of $secretname IS EXPORTED TO SECRETS TEXT FILE"
            }
          else {
          $secretname = az keyvault secret list --vault-name $(KVName) --query [$i].name -o tsv
          echo "There are other TAG(S) configured apart from Tag "$(KVTag)" for $secretname"
          }
          } 
```

| __IMPORTANT NOTE:-__ |
| --------- |
| You can comment and uncomment the Task based on your requirement (__Use Case #1 or Use Case #2__). For this Blog, I am focusing on __Exporting Secret with Specific Defined Tags [RequiredBy: CloudProvider] which is Use Case #2__ |

| __PIPELINE RESULTS:-__ |
| --------- |
| ![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/kj97tljrxs85z5iwbkok.png) |

| __SECRETS TEXT FILE PUBLISHED IN ARTIFACTS:-__ |
| --------- |

| ![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/2a457jflf5zc9qsxm8ms.png) |
| --------- |

| ![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/fl5ta6r7a3tz3tl1d9l6.png) |
| --------- | 

| __DOWNLOAD AND VIEW THE SECRETS TEXT FILE:-__ |
| --------- |

| ![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/m0csiesgp20ypvc0urm1.png) |
| --------- |
| The Output Format in Secrets Text file is [**NAME OF THE SECRET**]:[**VALUE OF THE SECRET**] |
| __As the Secrets AMSecret005 and AMSecret008 has similar tags [RequiredBy: CloudProvider], only those secrets with values got exported to Secrets.txt file. Rest of the Secrets AMSecret005 and AMSecret008 got ignored__  |