Delete Old Backup/Logfiles/Temp files from EFS using Lambda, PowerShell & Event Bridge
Tags: lambda, serverless, efs, aws
Published: 2022-07-09 07:04:47

Recently i was working on a requirement where a legacy application was migrated to AWS using Lift and Shift strategy.  This application used to generate lot of log data and Temporary files.

The log, temp and backup folder size used to reach around 20GB in 2 days, we were using EFS and there were multiple EC2 instances that were connected to EFS.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/dmbtbsv6ukr2obn0om9r.png)

After analyzing each file we decided that temporary files were just used for pre-processing and they were only required one time so we decided to delete those along with older log files (except access logs) & Old backup files.

I decided to use Powershell Lambda and integrate EFS with Lambda, this function will delete all un-necessary data from EFS. This function will execute periodically using Event Bridge & delete un-necessary files.

We will create Lambda function first using Visual Studio Code

Step 1 : Download Visual Studio Code
Step 2 : Execute below commands in powershell


`$region_code = 'ap-south-1'
$secretKey='your secret key'
$accessKey='your access key'`

`#Install-Module AWSPowerShell -Scope CurrentUser`
`#Install-Module AWSLambdaPSCore -Scope CurrentUser`

`import-Module AWSPowerShell -Force
Import-Module AWSLambdaPSCore -Force`

`#configure script credentials
Initialize-AWSDefaultConfiguration -AccessKey $accessKey -SecretKey $secretKey -Region ap-south-1
Set-AWSCredential -AccessKey $accessKey -SecretKey $secretKey -StoreAs default`

`#shows existing function list from AWS
Get-LMFunctionList`

`Get-AWSPowerShellLambdaTemplate`
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/gqyfo2iuhufssvg1g22x.png)

We'll be using Basic Barebone Script and adding our script commands to it

#Create a Starter Lambda function
`New-AWSPowerShellLambda -ScriptName DeleteUnusedFiles -Template Basic`

<u>You will get this project folder in explorer pane in your Visual Studio Code or else you can import the folder, this project gets created in the same </u>

Step 3 : Add below command to your Powershell script, this is to ensure that our Lambda function is successfully able to get 

`cd /mnt/efs
Remove-Item * -Include *.tmp 
ls /mnt/efs`

Step 4 : upload Lambda function to AWS


```
$publishPSLambdaParams = @{                                                                       
     name = "DeleteUnusedFiles "
     ScriptPath = ".\DeleteUnusedFiles\DeleteUnusedFiles.ps1"
     Region = "ap-south-1"
     IAMRoleArn = "delete_backup_rp_role"
     }

Publish-AWSPowerShellLambda @publishPSLambdaParams
```
  

 
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/bsobadpimcqkga0srr39.png)

I have created an EFS file system and Access points as below (You can refer below video for more information on setting up EFS and Mount Targets) :  {% embed https://www.youtube.com/watch?v=jhU7tX3D4ZE&t=1513s %}

Below are the details for my Accesspoint:

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/pqof49cb0worilqhph7e.png)

Once EFS is created and  Access points are ready, you need to login to your EC2 instance and mount EFS

1) install EFS Mount Helper

sudo yum install -y amazon-efs-utils

2) create folder as efs

mkdir /efs

3) Mount EFS

sudo mount -t efs -o tls,accesspoint=fsap-9999999999999999 fs-8888888888888888:/ efs

> Important Step

Remember to configure security group in EFS as below

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/0wn6w90hiworbdfv2u49.png)
 

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/k2w16frv5xgyh00hfcbp.png)
 
 
Once above steps are completed, we will integrate EFS with Lambda as below:

**** Important, you must associate Lambda to a VPC first, without doing this step, you wont be able to attach EFS

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/px3by2j0tn7qgf2v9def.png)
 
 
1) Create Role with below permissions and associate it with Lambda function
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/otqf625uahk2o0l6mskq.png)
 
2) Open your Lambda Function (In this example i am using PowerShell lambda function)
3) Navigate to Configuration tab 
4) Open File-System Subtab

Select EFS File System, Access point and Local Mount Point as per below screenshot and Hit save/create button.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/usjjnx4o6hukshd6ki6d.png)

Once successfully published you can test your function by opening your Lambda function and clicking on Test, after creating Test event.

As you can see in below screenshot we can see the result of ls command which shows files that are there in the EFS share and hence this shows that we can access the EFS from Lambda.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/9z1u7c6po4n5cjkvr1xw.png)

Once this is successfully created you can schedule your Lambda function using Event Bridge to execute periodically.

Troubleshooting:

If you get an error stating "Cannot configure .NET CLI", open below file from Path in Notepad++

C:\Program Files\WindowsPowerShell\Modules\AWSLambdaPSCore\3.0.1.0\Private\_DeploymentFunctions.ps1

Find for keyword $LASTEXITCODE
Comment all occurrence and conditions that are using this variable and run below commands

```
import-Module AWSPowerShell -Force
Import-Module AWSLambdaPSCore -Force
```

Try to upload Lambda function again using Powershell, it should work now.

If you have any other comment, suggestion for this post then do let me know.

Please do subscribe my YouTube channel https://www.youtube.com/c/Cloudperceptor/videos

Thanks,
Ujjwalkumar Soni