How to Distribute Secrets for PowerShell Scripts Using Ansible
Tags: ansible, powershell, windows
Published: 2020-04-17 05:47:40


# The Use Case
Managing secrets is hard. Everything needs to run under its own username/password, and apparently keeping plaintext passwords in scripts is really bad. 

To manage secrets better at our company we already implemented the following practices:
1) Avoiding hardcoding credentials (or any other configuration data) in the scripts, and commit scripts and configs separately.

2) Not actually storing real values in the config files: Instead, use placeholders like `#{sqlserverPassword}#` as values, and use a **CI Task** to replace the tokens with the actual values when deploying the script (many CI platforms have this feature out-of-the-box).

So even though we don't store passwords in Source Control anymore, when the scripts are deployed, the files may still contain passwords in plaintext.

You could use `Integrated Security` to avoid using passwords altogether, but this is only an option if your workers and resources are in the same **Windows Domain**. 
You can generate `SecureStrings` and have your script users use those instead. It's pretty secure because only users can decrypt the `SecureStrings` that the same user had created on the same machine. But the downside is only users can decrypt the `SecureStrings` that the same user had created on the same machine 😅

So how to solve this?

First, we need a method for adding and retrieving secrets: Use Microsofts new [SecretsManagement Module](https://devblogs.microsoft.com/powershell/secrets-management-development-release/).

> **Note**: The Module is still in PreRelease, but it can Add Secrets and Get Secrets, so good enough for our use case for now

So what is this Module?
> The Secrets Management module helps users manage secrets by providing a set of cmdlets that let you store secrets locally, using a local vault provider, and access secrets from remote vaults. This module supports an extensible model where local and remote vaults can be registered and unregistered on the local machine, per user, for use in accessing and retrieving secrets.

Notice that it says *per user*: Users can only access their own local vault (assuming we are not using external vaults like **Azure KeyVault**, which probably would make this whole tutorial unnessecary). 
To have the password available for the user to use, we must perform an `Add-Secret` command under the  user context, so that the user can do `Get-Secret` to retrieve the password stored in their own vault. 
The workflow would be something like: Invoke a command with the User Credentials passed as `-Credential`, executing a Scriptblock `{ Add-Secret -Name MySecret -Secret SuperSecretPassword }` .

🚦 *If there is a better/different way to provide secrets under different user contexts, please let me know.
For now, we continue this route.* 🚦
# The PowerShell Script to Add Secrets
I tried many methods to perform commands under a different user context:
- use `Invoke-Command -ScriptBlock $Scriptblock -Credential $creds`
  - why not? This requires **WinRM** to be available for the given user, even if the command is run on `localhost`, so I looked for something else.
- the `Start-Job -ScriptBlock $Scriptblock -Credential $creds | Wait-Job | Receive-Job` Combo
  - why not? This worked pretty consistently, up until to point I tried to integrate this in a CI Task. I got stuck getting ❌`2100,PSSessionStateBroken` errors, and it seems to be a [common issue](https://issues.jenkins-ci.org/browse/JENKINS-49159) in CI systems. Apparently it has something to do with credentials not being passed while 'double hopping', resolving this would be again setting up WinRM in conjunction with something called CrepSSP.

So finally I ended up using the [Invoke-CommandAs](https://github.com/mkellerman/Invoke-CommandAs) Module, which under the hood creates a **Scheduled Task** as the user, carries out your commands you specify in a `$ScriptBlock`, and finally removes the Task.
An example snippet for adding 1 secret for 1 user on 1 machine would be:
```
$userName = "domain\user_account"
$userPassword = "userPassword" # ✋Warning If the password contains $ signs, use single quotes!        
[pscredential]$credObject = New-Object System.Management.Automation.PSCredential ($userName,         $userPassword)            
$Credentials = Get-Credential $credObject 
$ScriptBlock = {          
   If ( -not(Get-Module Microsoft.PowerShell.SecretsManagement -listAvailable) ) {
      Write-Host "Secret Module not found, installing.."
      [Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12
      Install-Module -Name Microsoft.PowerShell.SecretsManagement -RequiredVersion 0.2.0-alpha1 -AllowPrerelease -Repository psgallery -Force -Scope CurrentUser
   }
   # proof that it worked:
   Write-Host $env:computername
   Write-Host $env:username
   Get-SecretInfo
}
Invoke-CommandAs -Scriptblock $ScriptBlock -AsUser $Credentials
```

> **Note**: Notice the Tls Securtity protocol, I almost flipped my desk when I couldn't figure out why the Module wouldn't download, until I saw [this](https://devblogs.microsoft.com/powershell/powershell-gallery-tls-support/)

**Problem**: The user needs the right amount of user rights permissions to pull this off. I haven't fully figured out the exact set of permissions, I thought the user only needs the local `log on as a batch job` permission, but couldn't get this to work consistenly. 
 **Solution**: So I decided to throw this hack in for now: I ended up temporaly adding the user to the local `Administrators` group, add the secrets, and then remove it again from the group.
```
NET LOCALGROUP "Administrators" $userName /ADD | Out-Null
Invoke-CommandAs -Scriptblock $ScriptBlock -AsUser $Credentials
NET LOCALGROUP "Administrators" $userName /remove| Out-Null
##📣 If anyone has a better idea, please let me know :halp: 📣
```

🎉 Great, now that we have cobbled together a working script, we can now use this for every secret, for each user, on every machine that the user operates on, right? Also, apparantly current security conventions suggest that we should run each service as a different user, therefore increasing the amount users we have to manage 🤯. You might see how this can be a bit tedious to manage manually.
But here comes **Ansible**.
# The  Ansible Playbook

> Ansible is a radically simple IT automation engine that automates [cloud provisioning](https://www.ansible.com/provisioning?hsLang=en-us), [configuration management](https://www.ansible.com/configuration-management?hsLang=en-us), [application deployment](https://www.ansible.com/application-deployment?hsLang=en-us), [intra-service orchestration](https://www.ansible.com/orchestration?hsLang=en-us), and many other IT needs.

If you haven't played around with Ansible yet, I suggest watching the [live-streams](https://www.youtube.com/watch?v=goclfp6a2IQ&list=PL2_OBreMn7FplshFCWYlaN2uS8et9RjNG&index=4) of this Jeff Geerling guy or check out his [Ansible book](https://leanpub.com/ansible-for-devops) (which is currently free)

> NOTE: If there are more 'ansible' ways to achieve this use-case, please let me know.

The highlevel workflow would be:
- hosts: list of servers
- variables:
  - `accounts`: 
   a dictionary where key = accountname, value = password
  - `account_mapped_secrets`:
  a dictionary where  key = accountname, value= list of secret-keys
  - `secrets`:
  a dictionary where key = secret-key, value=secret-value

- playbook:
  - Get list of `accounts` that have secrets mapped
  - Loop over  `accounts`, and for each `account`
    - construct a secret key/value disctionary for each `account_secret` of the `account`
    - use the above `Invoke-CommandAs` powershell snippet under the `account` user context to add each secret-key/value

A play would look something like this:
- the `main.yml` file
```
- name: Loop over Accounts that has Secrets to deployed
  include_tasks: add_secrets.yml  
  with_items: "{{ account_mapped_secrets }}"  
  loop_control:  
    loop_var: account  
    extended: yes
```
- the included tasks `add_secrets.yml` file
```
---  
- name: "${{ account }}$ -- Create temp Vault dict"  
  set_fact:  
    account_secrets_{{ ansible_loop.index }}: {}  
  
- name: "${{ account }}$ -- Populate Secrets"  
  set_fact:  
   account_secrets_{{ ansible_loop.index }}: "{{ lookup('vars', 'account_secrets_' ~ ansible_loop.index) |default({}) | combine( {item: secrets[item]} ) }}"  
  with_items: "{{ account_mapped_secrets[account] }}"  
  
- name: "${{ account }}$ -- Add Secrets to Local Vault"  
  win_shell: |  
   $userName = '{{ account }}'  
   $userPassword = '{{ accounts[account] }}'  
   [securestring]$secStringPassword = ConvertTo-SecureString $userPassword -AsPlainText -Force  
   [pscredential]$credObject = New-Object System.Management.Automation.PSCredential ($userName, $secStringPassword)  
  
   $ScriptBlock = {  
  
   Get-SecretInfo | Remove-Secret -Vault BuiltInLocalVault  #✋for demo purposes we delete any existing Secrets.
   $json = @"  
   {{ lookup('vars', 'account_secrets_' ~ ansible_loop.index) | to_nice_json }}  
   "@  
   ($json | ConvertFrom-Json).psobject.properties | ForEach-Object { Add-Secret -Name $_.Name -  Secret $_.Value }  
   Get-SecretInfo  
  
   }  
   try {  
     NET LOCALGROUP "Administrators" $userName /ADD | Out-Null  
     Invoke-CommandAs -ScriptBlock $ScriptBlock -AsUser $credObject -verbose  
     NET LOCALGROUP "Administrators" $userName /DELETE | Out-Null  
   } catch {  
     Write-Error $_  
   }  
  register: shellresult  
  
- name: "${{ account }}$ -- Fail if output is not exptected"  
  fail:  
    msg: shellresult.stdout_lines  
  when: shellresult.stdout.find("Vault") == -1  
  
- name: "${{ account }}$ -- Show Errors"  
  fail:  
    msg: "{{ shellresult.stderr }}"  
  when: shellresult.stderr != ""  
  
- name: "${{ account }}$ -- Print Result"  
  debug:  
    msg: "{{ shellresult.stdout_lines }}"
```

✋ You need to setup an [Ansible Role](https://docs.ansible.com/ansible/latest/user_guide/playbooks_reuse_roles.html) to make this snippet work as it is.
My directory looks someting like this:
```
/etc/ansible  
├── ansible.cfg  
├── environs  
│ ├── dev                     # copy this subfolder for each environment (dev,test,prod...) you might have  
│ │ ├── group_vars  
│ │ │ └── all  
│ │ │      ├── main.yaml      # contains account_mapped_secrets variable
│ │ │      └── accounts.yaml  # contains accounts variable
| | |      └── secrets.yaml   # contains secrets variable
│ │ ├── hosts  
....
├── roles  
│ ├── secrets  
│ │ ├── tasks  
│ │ │ ├── add_secrets.yml     # the core of the play
│ │ │ └── main.yml            # the main tasks file that includes the above file in a loop
├── play_secrets_role.yaml    # a play that calls the secrets role
```
To run this play for the `dev` environment, run: `ansible-playbook -i environs/dev play_secrets_role.yaml`

## Explaining the playbook:
- You can notice that I'm construcing an unique dictionary in each loop `account_secrets_{{ ansible_loop.index }}`. In the first versions of my playbook I just used `account_secrets` as the variable, and apparantly in Ansible, once you set a fact, you can't unset it. So the secrets dictionary would just keep appending between each loop, and the last user would end up having all the secrets.

-  `ansible_loop.index` is the current iteration in the loop, but is only available when you have the `extended: yes` option when looping.

- We use a jinja `combine()` expression to dynamically create the `account_secrets_x` dictionary. Note that I refer to the above variable with the `lookup('vars', 'account_secrets_' ~ ansible_loop.index)` syntax instead of the `account_secrets_{{ ansible_loop.index }}`. 
This is because the following won't work: `"{{ account_secrets_{{ ansible_loop.index }}) |defaul t({}) | combine( {item: secrets[item]} ) }}"` as you can't have double interpolation inside a jinja expression.

## Wait a minute, you're still storing plaintext passwords in the variable files!
Here comes Ansible again. By separate the `accounts` and `secrets` from the `account_mapped_secrets` into different variable files we can encrypt the first two with [Ansible Vault](https://docs.ansible.com/ansible/latest/user_guide/vault.html#file-level-encryption)
Then when running your `ansible-playbook`, pass in the extra argument `--ask-vault-pass`