An AWS Summer: EFS & Lambda + Serverless Framework
Tags: aws, serverless, tutorial, productivity
Published: 2020-10-05 09:56:18

<!-- wp:paragraph -->
<p>The autumn equinox has just passed, which is a perfect moment to look back, and review some of the features released in this last summer by AWS - in no particular order, just because I think they are cool - and useful :)</p>
<!-- /wp:paragraph -->

<!-- wp:image {"width":771,"height":415,"sizeSlug":"large"} -->
<figure class="wp-block-image size-large is-resized"><img src="https://upload.wikimedia.org/wikipedia/commons/8/8b/North_season.jpg" alt="" width="771" height="415"/></figure>
<!-- /wp:image -->

<!-- wp:paragraph -->
<p></p>
<!-- /wp:paragraph -->

<!-- wp:heading -->
<h2>Serverless challenges </h2>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>If you´ve been developing serverless applications for a while, pretty sure you have found yourself facing a few challenges, apart from the <em>old cold start thing</em> - which have been solved to a great extent with the <em>Provisioned Concurrency</em> feature. </p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>For instance, let's say you need to load large files of rules consumed by a Lambda function, that implements a rules engine, or you need to keep data files produced dynamically by the function between invocations. Lambda provides some local space - 512MB - that you may use, but it's small and ephemeral, so is not useful for those kinds of scenarios.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Other solutions come to mind: storing in databases - <em>RDS, DynamoDB,S3 </em> ... but comes with a high price of development, performance and cost. What would happen if we had peaks of several hundreds - or thousands requests - per second,  loading big files in the startup and writing files to a data store concurrently? </p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Well, at the very least, we could have a big performance hit, depending on the size of the files, the latency of retrieving the files at startup + the cold start of Lambdas - <em>enter provisioned concurrency</em> - plus the latency of storing the intermediate files to the datastores - it's not the same storing and retrieving from S3 than from DynamoDB.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>So no alternative? Well, we are in luck, as AWS released <a href="https://aws.amazon.com/es/efs/">EFS</a> support for Lambda in June!</p>
<!-- /wp:paragraph -->

<!-- wp:image {"id":537,"width":743,"height":216,"sizeSlug":"large"} -->
<figure class="wp-block-image size-large is-resized"><img src="https://metanube.files.wordpress.com/2020/09/image.png?w=1024" alt="" class="wp-image-537" width="743" height="216"/><figcaption>Image property of AWS</figcaption></figure>
<!-- /wp:image -->

<!-- wp:paragraph -->
<p>Amazon <a href="https://aws.amazon.com/es/efs/">EFS</a> is widely known, so I'm not going to delve depth into the service, but just to mention that <em>Amazon</em> <em>Elastic File Service</em> provides a NFS file system that escalates on demand, providing high throughput and low latency. It's very useful when shared storage, and a parallel access from the services it´s needed.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p></p>
<!-- /wp:paragraph -->

<!-- wp:heading {"level":3} -->
<h3>Configuration &amp; Considerations</h3>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>"With power comes responsibility", or in our case with powerful features come some configuration constraints. EFS runs in different subnets within a VPC, which means that our Lambda functions have to run within a VPC as well. That comes with a price: IP directioning, possible performance hit, loss of connection to AWS global services, therefore a <em>NAT Gateway or Private Links / Gateway</em> might need to be used, depending on the use case.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>That constraint was vastly improved last year when <a href="https://aws.amazon.com/blogs/compute/announcing-improved-vpc-networking-for-aws-lambda-functions/">Hyperplane ENI for Lambda</a> was released, allowing that just a few ENI´s  - and therefore a few IP´s - would be enough to handle a big number of Lambda invocations, decoupling function scaling from ENI´s provisioning.</p>
<!-- /wp:paragraph -->

<!-- wp:heading {"level":4} -->
<h4>Configuration - Serverless Framework</h4>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>The configuration of a Lambda function running within a VPC could be fairly simple - if only needs to access the VPC resources - as in shown in the image below - under the vpc label:</p>
<!-- /wp:paragraph -->

<!-- wp:image {"align":"center","id":548,"width":569,"height":381,"sizeSlug":"large"} -->
<div class="wp-block-image"><figure class="aligncenter size-large is-resized"><img src="https://metanube.files.wordpress.com/2020/09/image-5.png?w=938" alt="" class="wp-image-548" width="569" height="381"/><figcaption>Serverless framework YAML - Image  MNube.org</figcaption></figure></div>
<!-- /wp:image -->

<!-- wp:paragraph -->
<p>A security group is needed for the Lambda function, the ID´s of subnet(s) where the ENI(s) will be placed, and permissións to <em>create, delete, and describe network interfaces</em>.</p>
<!-- /wp:paragraph -->

<!-- wp:image {"align":"center","id":590,"width":1008,"height":288,"sizeSlug":"large"} -->
<div class="wp-block-image"><figure class="aligncenter size-large is-resized"><img src="https://metanube.files.wordpress.com/2020/10/vpc-1.png?w=1024" alt="" class="wp-image-590" width="1008" height="288"/><figcaption>VPC Lambda - Image MNube.org</figcaption></figure></div>
<!-- /wp:image -->

<!-- wp:paragraph -->
<p></p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>The Lambda function is running within our VPC now, <em>an ENI placed in each subnet selected</em>, but in order to access the EFS instance a few permissións will need to be provided:</p>
<!-- /wp:paragraph -->

<!-- wp:image {"id":558,"width":538,"height":357,"sizeSlug":"large"} -->
<figure class="wp-block-image size-large is-resized"><img src="https://metanube.files.wordpress.com/2020/09/image-9.png?w=868" alt="" class="wp-image-558" width="538" height="357"/><figcaption>Role permissións EFS, Lambda - Image MNube.org</figcaption></figure>
<!-- /wp:image -->

<!-- wp:paragraph -->
<p>Now the EFS can be created within the VPC. In order to do that, the console, Cloudformation, Serverless, AWS CLI, AWS SDK, etc ... could be used.</p>
<!-- /wp:paragraph -->

<!-- wp:image {"align":"center","id":592,"sizeSlug":"large"} -->
<div class="wp-block-image"><figure class="aligncenter size-large"><img src="https://metanube.files.wordpress.com/2020/10/vpc-2.png?w=1024" alt="" class="wp-image-592"/><figcaption>EFS instance - Image MNube.org</figcaption></figure></div>
<!-- /wp:image -->

<!-- wp:paragraph -->
<p>After creating the instance, an <em>access point</em> needs to be provided to allow applications access. This is a new resource: <em>"AWS::EFS::AccessPoint"</em>. It can be created from the console, or through a cloudformation file - we will need to supply the EFS ID: ${self.provider}.</p>
<!-- /wp:paragraph -->

<!-- wp:image {"id":545,"width":571,"height":353,"sizeSlug":"large"} -->
<figure class="wp-block-image size-large is-resized"><img src="https://metanube.files.wordpress.com/2020/09/image-4.png?w=992" alt="" class="wp-image-545" width="571" height="353"/><figcaption>Serverless framework YAML - Image MNube.org</figcaption></figure>
<!-- /wp:image -->

<!-- wp:paragraph -->
<p>Finally, we link the file system to the Lambda Function, providing the arn of the EFS, the arn of the access point, and the local mounted path - as shown on the image below:</p>
<!-- /wp:paragraph -->

<!-- wp:image {"align":"center","id":595,"sizeSlug":"large"} -->
<div class="wp-block-image"><figure class="aligncenter size-large"><img src="https://metanube.files.wordpress.com/2020/10/vpc-3-1.png?w=1024" alt="" class="wp-image-595"/><figcaption>Image MNube.org</figcaption></figure></div>
<!-- /wp:image -->

<!-- wp:paragraph -->
<p>The EFS instance is ready to be accessed by the Lambda function :)</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p></p>
<!-- /wp:paragraph -->

<!-- wp:heading {"level":3} -->
<h3>Solution</h3>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>I have used the <em><strong><a href="https://www.serverless.com/open-source/">Serverless framework</a></strong></em> to produce the solution - but  <em><strong>A</strong></em><strong><em>WS SAM</em> <em>with Cloud 9</em></strong> as the official alternative could have been used instead. I have quite experience with Serverless, having introduced it to a few companies - <em>including <a href="https://www.everis.com/global/en">Everis</a></em> - with big success.</p>
<!-- /wp:paragraph -->

<!-- wp:image {"id":605,"sizeSlug":"large"} -->
<figure class="wp-block-image size-large"><img src="https://metanube.files.wordpress.com/2020/10/aws-lambda-efs-support.png?w=1024" alt="" class="wp-image-605"/><figcaption>Architecture - MNube.org</figcaption></figure>
<!-- /wp:image -->

<!-- wp:paragraph -->
<p>Let's create - or transfer -  a rules file that can be accessed from the Lambda function :)</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Different services could be used to transfer the files, like <em>AWS DataSync</em>, an <em>EC2</em> instance, or even creating files from code. The files we might transfer from EC2 are accessible from the Lambda functions, so we´ll use this method.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>After the EC2 instance has been created - <em>a t2.micro is enough</em>  - in one of the subnets of the VPC that has access to the <em>EFS ENI´s</em>, a directory we´ll be needed -  <strong>/<em>efs</em></strong>. That directory doesn't have any link to the EFS instance, so we´ll need to mount the directory. </p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>One way to do it is using the <em>EFS tools</em>:</p>
<!-- /wp:paragraph -->

<!-- wp:code -->
<pre class="wp-block-code"><code>                     sudo yum install -y amazon-efs-utils</code></pre>
<!-- /wp:code -->

<!-- wp:paragraph -->
<p>An <em>access point</em> was created previously that we can use to mount the directory. It's easy to get the command line needed from the web console. Just go to to the <em>Amazon EFS &gt; Access Point &gt; id</em> link, and press the <em><strong>Attach</strong></em> button:</p>
<!-- /wp:paragraph -->

<!-- wp:image {"id":597,"sizeSlug":"large"} -->
<figure class="wp-block-image size-large"><img src="https://metanube.files.wordpress.com/2020/10/vpc-4.png?w=1024" alt="" class="wp-image-597"/><figcaption>EFS Mount - Image MNube.org</figcaption></figure>
<!-- /wp:image -->

<!-- wp:paragraph -->
<p>After mounting the directory - <em>in green </em>- the files can be transfer to the<em> <strong>/efs</strong></em> directory:</p>
<!-- /wp:paragraph -->

<!-- wp:image {"id":599,"sizeSlug":"large"} -->
<figure class="wp-block-image size-large"><img src="https://metanube.files.wordpress.com/2020/10/vpc-5.png?w=1024" alt="" class="wp-image-599"/><figcaption>Mounting and creating files - Image MNube.org</figcaption></figure>
<!-- /wp:image -->

<!-- wp:paragraph -->
<p>At this point, the access to the directory from the Lambda function should be fully possible. I have coded a minimum Lambda function that lists the files contained in the directory:</p>
<!-- /wp:paragraph -->

<!-- wp:image {"id":587,"width":536,"height":387,"sizeSlug":"large"} -->
<figure class="wp-block-image size-large is-resized"><img src="https://metanube.files.wordpress.com/2020/10/image-7.png?w=970" alt="" class="wp-image-587" width="536" height="387"/><figcaption>Lambda function - Image MNube.org</figcaption></figure>
<!-- /wp:image -->

<!-- wp:paragraph -->
<p>The solution is now ready to be deployed. Keep in mind that I have only shown parts of the <em>serverless.yml</em>, equivalent to the cloudformation file you might use to provide the infrastructure - I will leave that to you as an exercise.</p>
<!-- /wp:paragraph -->

<!-- wp:syntaxhighlighter/code -->
<pre class="wp-block-syntaxhighlighter-code">                serverless deploy --stage dev --region eu-west-1</pre>
<!-- /wp:syntaxhighlighter/code -->

<!-- wp:image {"id":600,"sizeSlug":"large"} -->
<figure class="wp-block-image size-large"><img src="https://metanube.files.wordpress.com/2020/10/vpc-6.png?w=1024" alt="" class="wp-image-600"/><figcaption>Serverless Stack - Image MNube.org</figcaption></figure>
<!-- /wp:image -->

<!-- wp:paragraph -->
<p>An URL link is provided by the framework, as I created an API gateway that invokes the Lambda function:</p>
<!-- /wp:paragraph -->

<!-- wp:image {"align":"center","id":586,"width":1100,"height":261,"sizeSlug":"large"} -->
<div class="wp-block-image"><figure class="aligncenter size-large is-resized"><img src="https://metanube.files.wordpress.com/2020/10/image-6.png?w=1024" alt="" class="wp-image-586" width="1100" height="261"/><figcaption>Cloudwatch Logs - Image from MNUBE.org</figcaption></figure></div>
<!-- /wp:image -->

<!-- wp:paragraph -->
<p>I have captured the request trace from the Cloudwatch Logs, where we can see the files in /efs: <em><strong>test.txt and rules.txt</strong></em>, and the low latency of the request.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p></p>
<!-- /wp:paragraph -->

<!-- wp:heading {"level":3} -->
<h3>Other Use Cases</h3>
<!-- /wp:heading -->

<!-- wp:list -->
<ul><li>Loading big libraries that Lambda layers can´t handle.</li><li>Files that are updated regularly.</li><li>Files that need locks for concurrent access. </li><li>Access to big files - zip / unzip.</li><li>Using different computing architectures - <em>EC2, ECS</em> -  to process the same files.</li></ul>
<!-- /wp:list -->

<!-- wp:paragraph -->
<p></p>
<!-- /wp:paragraph -->