Deploying AWS Serverless — a plague o’ both your houses!
Tags: aws, serverless
Published: 2020-05-19 01:06:33

[Uclusion](https://www.uclusion.com/?utm_source=devto&utm_medium=blog&utm_campaign=devdeployawsserverless) had what should have been a simple problem. We needed to deploy our Lambda, DynamoDB and React based application to dev, stage and production environments without owning a lot of our own code to do so. The code is spread out over many repositories where each repo exposes a different set of back end APIs or holds the front end code. The back end APIs are testing by automated integration tests and the front end by a combination of automated and manual QE. In addition sometimes upgrade scripts must be run before deploying.

We expected to adopt some out of the box solution in CircleCI but instead ended up having to write a lot of code to tag the repos that passed the integration tests as they go from one environment to the next. Here is a partial excerpt from that code:

    from utils.constants import rest_api_backend_repos
    import sys
    
    
    def get_latest_release_with_prefix(releases, prefix):
        latest = None
        latest_date = None
        for release in releases:
            if release.tag_name.startswith(prefix):
                created_at = release.created_at
                if latest is None or created_at > latest_date:
                    latest = release
                    latest_date = created_at
        return latest
    
    
    def find_latest_release_with_prefix(repo, prefix):
        releases = repo.get_releases()
        latest_release = get_latest_release_with_prefix(releases, prefix)
        if not latest_release:
            print("Couldn't get a release with prefix " + prefix + " for " + repo.name)
            sys.exit(3)
        return latest_release
    
    
    def get_latest_releases_with_prefix(github, prefix, repo_name=None, is_ui=False):
        if repo_name:
            repos_to_search = [repo_name]
        else:
            repos_to_search = rest_api_backend_repos if not is_ui else ['uclusion_web_ui']
        candidates = []
        for repo in github.get_user().get_repos():
            if repo.name in repos_to_search:
                latest_release = find_latest_release_with_prefix(repo, prefix)
                candidates.append([repo, latest_release])
        if len(candidates) != len(repos_to_search):
            print("Some repos are missing tags")
            sys.exit(5)
        return candidates
    
    
    def get_tag_for_release(repo, release):
        for candidate in repo.get_tags():
            if candidate.name == release.tag_name:
                return candidate
        return None
    
    
    def get_tag_for_release_by_repo_name(github, repo_name, release):
        for repo in github.get_user().get_repos():
            if repo.name == repo_name:
                return get_tag_for_release(repo, release)
        return None
    
    
    def clone_release(repo, old_release, new_name):
        # get the tag for the release
        tag = get_tag_for_release(repo, old_release)
        if not tag:
            sys.exit(4)
        sha = tag.commit.sha
        repo.create_git_tag_and_release(new_name, 'Blessed build tag', new_name, 'Blessed', sha, 'commit')
    
    
    def get_master_sha(github, repo_name):
        for repo in github.get_user().get_repos():
            if repo.name == repo_name:
                head = repo.get_git_ref('heads/master')
                return head.object.sha
        return None
    
    
    def release_head(github,dest_tag_name, prebuilt_releases, repo_name=None, is_ui=False):
        sha_map = {}
        for entry in prebuilt_releases:
            repo = entry[0]
            release = entry[1]
            tag = get_tag_for_release(repo, release)
            if tag:
                sha_map[repo.name] = tag.commit.sha
    
        if repo_name:
            repos_to_search = [repo_name]
        else:
            repos_to_search = rest_api_backend_repos if not is_ui else ['uclusion_web_ui']
    
        for repo in github.get_user().get_repos():
            if repo.name in repos_to_search:
                head = repo.get_git_ref('heads/master')
                sha = head.object.sha
                if sha != sha_map[repo.name]:
                    print("Will clone head of " + repo.name + " to " + dest_tag_name)
                    repo.create_git_tag_and_release(dest_tag_name, 'Head Build', dest_tag_name, 'Head', sha, 'commit')
                else:
                    print("Skipping " + repo.name + " because head has already built")
    
    
    def clone_latest_releases_with_prefix(github, source_prefix, dest_tag_name, repo_name=None, is_ui=False):
        print("Cloning releases")
        candidates = get_latest_releases_with_prefix(github, source_prefix, repo_name, is_ui)
        for candidate in candidates:
            repo = candidate[0]
            release = candidate[1]
            print("Will clone " + release.tag_name + " in repo " + repo.name + " to " + dest_tag_name)
            clone_release(repo, release, dest_tag_name)

and then more code to run the upgrade scripts.

The situation does not seem any better in Github Actions because [blogs like this](https://blog.marcnuri.com/triggering-github-actions-across-different-repositories/) are being written and feature requests [like this](https://github.community/t5/GitHub-Actions/GitHub-Actions-Manual-Trigger-Approvals/td-p/31504) have a horde of beggars.

So how did our out of the box Serverless deployment dream get deferred? Should we, as [SEED](https://seed.run) suggests, use a monorepo? **HELL NO**.

Uclusion of course has code that is shared between back end services and we put those in [Lambda layers](https://docs.aws.amazon.com/lambda/latest/dg/configuration-layers.html) (which sadly have their own order of deployment and upgrade scripts). But there is no way we went to all the trouble of using Lambdas just to have them turn to spaghetti in a giant shared repository. There has to be a better way then jumping from the frying pan into the fire.

Could the problem be that we don’t have a complex enough branching strategy? For instance the creators of Serverless configuration YAML (which we use) [describe here](https://www.serverless.com/learn/guides/cicd/) a three AWS account dev, stage and prod structure like ours but also say:
> Promoting from staging to production
> If everything in staging looks ok and ready to promote to production, we’ll use our git workflow to merge the desired commit from master to the prod branch. Since we already configured branch deployments for the prod branch, it will automatically run the tests and deploy to the prod stage.

Let’s break down this paragraph though. “If everything in staging looks ok” — that’s got to be passing integration tests that run across all of your services. There is no way any one service has sufficient testing to promote it on its own and at the same time require ***staging*** with all of the other services. But then what is the “desired commit” that you are supposedly merging? Your integration tests ran across many commits over many repositories unless you use a monorepo or again aren’t really staging anything at all.

So in conclusion I can’t offer you any out of the box solution to what should have been a generic solved problem. I can only give you some small comfort that you are not alone.