Scraping AWS Docs to Generate PowerShell Help in VaporShell
Tags: aws, powershell, documentation, cloudformation
Published: 2019-10-03 00:50:25

Recently, I became interested in what code repositories were automatically generated from the [AWS CloudFormation Resource Specification](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/cfn-resource-specification.html) files. These act as schema in how CloudFormation templates must be built.

Tools that came to mind as already likely doing so were the ones meant to help with CFN template development by abstracting it away from raw JSON/YAML and into a particular language like [SparkleFormation](https://www.sparkleformation.io/), [Troposphere](https://github.com/cloudtools/troposphere), and probably the [AWS Cloud Development Kit](https://github.com/aws/aws-cdk).

## VaporShell

![Alt Text](https://thepracticaldev.s3.amazonaws.com/i/pdmh50xu4wbjv6nbij89.png)

I have used PowerShell quite a bit throughout my tech career, and back when I first saw the CFN resource specs, I thought it would be cool to autogen a PowerShell Module that had the same goal in mind as those other tools. I also wanted to auto-populate the comment-based help within each function, by pulling directly from the AWS CloudFormation User Guide documentation. Then I shelved it and moved on to other things. This was in 2017.

It came to mind again recently, and I found that [VaporShell](https://vaporshell.io/) had been created in the meantime! I checked to see how the help documentation looked, and realized I could revisit my old code and contribute to the project.

### Example Improvements

Originally, the built-in comment-based help for VaporShell resource functions were barebones. I felt that the [AWS CloudFormation Resource Guide source](https://github.com/awsdocs/aws-cloudformation-user-guide) could be used to populate parameter and function documentation.

#### Parameter Information (VaporShell Original)

```
PS> Get-Help New-VSAlexaAskSkill -Parameter VendorId

-VendorId <Object>
    Required: True    
    Documentation: http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-ask-skill.html#cfn-ask-skill-vendorid   
    PrimitiveType: String    
    UpdateType: Immutable
    
    Required?                    true
    Position?                    named
    Default value                
    Accept pipeline input?       false
    Accept wildcard characters?  false
```

#### Parameter Information (Expanded)

Includes parsed data from [Alexa::ASK::Skill](https://github.com/awsdocs/aws-cloudformation-user-guide/blob/master/doc_source/aws-resource-ask-skill.md)

```
PS> Get-Help New-VSAlexaAskSkill -Parameter VendorId

-VendorId <Object>
    The vendor ID associated with the Amazon developer account that 
    will host the skill. Details for retrieving the vendor ID are in 
    How to get your vendor ID: https://github.com/alexa/alexa-smartho
    me/wiki/How-to-get-your-vendor-ID. The provided LWA credentials 
    must be linked to the developer account associated with this 
    vendor ID.

    Documentation: http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-ask-skill.html#cfn-ask-skill-vendorid   
    PrimitiveType: String    
    UpdateType: Immutable
    
    Required?                    true
    Position?                    named
    Default value                
    Accept pipeline input?       false
    Accept wildcard characters?  false
```

> _NOTE: `Required` property can be dropped, because PowerShell help output will automatically check whether `[parameter(Mandatory = $true)]` which results in `Required?` output being `true` or `false`._

#### Description Information (VaporShell Original)

```
PS> Get-Help New-VSAlexaAskSkill

NAME
    New-VSAlexaAskSkill
    
SYNOPSIS
    Adds an Alexa::ASK::Skill resource to the template
...

DESCRIPTION
    Adds an Alexa::ASK::Skill resource to the template
...
```

#### Description Information (Expanded)

Includes parsed data from [Alexa::ASK::Skill](https://github.com/awsdocs/aws-cloudformation-user-guide/blob/master/doc_source/aws-resource-ask-skill.md)

```
PS> Get-Help New-VSAlexaAskSkill

NAME
    New-VSAlexaAskSkill
    
SYNOPSIS
    Adds an Alexa::ASK::Skill resource to the template
...

DESCRIPTION
    The Alexa::ASK::Skill resource creates an Alexa skill that 
    enables customers to access new abilities. For more information 
    about developing a skill, see the Build Skills with the Alexa 
    Skills Kit developer documentation: https://developer.amazon.com/
    docs/ask-overviews/build-skills-with-the-alexa-skills-kit.html.
```

### How?

The best path seemed to be:

- Pull down the latest version of the AWS CFN User Guide
- Pull down the latest `us-east-1` CFN spec file
- Roll through the `Documentation` properties of everything within the AWS CFN spec files that are used to build PowerShell functions, and find the equivelant MarkDown doc in the AWS CFN User Guide source repo
- Scrape resource/property descriptions, along with any parameter descriptions, to include in PowerShell function comment-based help

#### Proof-of-Concept Output

Using the POC code I made here, [AWSCFNDocHelpers.psm1](https://gist.github.com/ScriptAutomate/28712f09f5726394e55e5790d8c8ac62), we can get exactly what we are looking for.

##### Prerequisites

- Tested On: PowerShell Core 6.2.2
- OS: Ubuntu 18.04
- Requires the following done in the `$PWD` code will execute in

```powershell
# AWS CFN User Guide help documentation
git clone https://github.com/awsdocs/aws-cloudformation-user-guide.git

# AWS CFN spec file
Invoke-WebRequest 'https://d1uauaxba7bl26.cloudfront.net/latest/gzip/CloudFormationResourceSpecification.json' -OutFile 'CloudFormationResourceSpecification.json'
```

##### Example Code Results

```powershell
PS> Import-Module ./AWSCFNDocHelpers.psm1
PS> $HelpDocs = New-CFNHelpDoc
PS> ($HelpDocs | Get-Member -Type NoteProperty).Name
Description
FunctionName
Links
Parameters
Synopsis

PS> $HelpDocs[0].FunctionName
New-VSAlexaASKSkill

PS> $HelpDocs[0].Description 
The Alexa::ASK::Skill resource creates an Alexa skill that enables customers to access new abilities. For more information about developing a skill, see the Build Skills with the Alexa Skills Kit developer documentation: https://developer.amazon.com/docs/ask-overviews/build-skills-with-the-alexa-skills-kit.html.

PS> $HelpDocs[0].Parameters[0].ParameterName
AuthenticationConfiguration

PS> $HelpDocs[0].Parameters[0].ParameterDescription
Login with Amazon LWA configuration used to authenticate with the Alexa service. Only Login with Amazon clients created through the Amazon Developer Console: https://developer.amazon.com/lwa/sp/overview.html are supported. The client ID, client secret, and refresh token are required.
```

VaporShell now automatically includes these changes, dynamically populating content scraped from the AWS CFN docs! Nate Ferrell owns the repository, and modified my demo code to be part of the CI pipeline:

**Nate Ferrel**

- [GitHub: @scrthq](https://github.com/scrthq)
- [Twitter: @scrthq](https://twitter.com/scrthq)

> ***NOTE:*** _To see the back-and-forth issue communication in the VaporShell repo, along with supplied demo code, checkout the issue here: https://github.com/scrthq/VaporShell/issues/61_

Working on this made me realize there was some funny business going on with the AWS CFN documentation, and with the CFN resource specification files. I'll talk about these problems, and how I used GitHub Actions with Python to do some automated auditing, in another post.

In the meantime, checkout the [GitHub Actions beta](https://github.com/features/actions)!

----

_Originally published at [https://icanteven.io](https://www.icanteven.io/blog/cfn-specs-vaporshell/) on October 2nd, 2019_