👨‍💻👩‍💻How to use AWS Systems Manager Automation Document🧐to fix✅SSH issue on a Linux EC2 Instance☁️📢
Tags: aws, linux, devops
Published: 2020-07-20 00:44:41

<b>When you face SSH issues over EC2 Instance👨‍💻☁️, What procedure do you follow ?</b>

![SSH issue](https://media.giphy.com/media/3oEjHGJE2H5KMDjyj6/giphy.gif)

>  - ***Stop the Instance --> detach root volume --> attach volume to rescue    Instance --> and then further steps to troubleshoot issue***

<b>Right ?🤔💭<b>

### <b>Here's something to Automate this procedure⏩⏩</b>
>  - <i>"This procedure will save a lot  of time which we invest with manual process. AWS Systems Manager Automation Document - **["AWSSupport-ExecuteEC2Rescue"](https://console.aws.amazon.com/systems-manager/automation/execute/AWSSupport-ExecuteEC2Rescue)** does all  the steps for you in an automated fashion."</i>
<br>

### <b>How it works ?👨‍🏭</b>
>  - It uses **[EC2Rescue for Linux](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/Linux-Server-EC2Rescue.html)** &  **[EC2Rescue for Windows](https://docs.aws.amazon.com/AWSEC2/latest/WindowsGuide/Windows-Server-EC2Rescue.html)** tool, which is a framework for executing diagnostic and troubleshooting modules for analyzing and remediating issues on **Windows and Linux instances.**

<code>**Here I will discuss about  most common SSH issue on Linux instance.**</code>

***
<i><b>Also, I will guide you through Console way to get familiar with the workflow. If you're interested in using AWS CLI, please check out my [Blog Post](https://dev.to/dineshrathee12/via-aws-cli-automating-my-post-using-aws-systems-manager-automation-document-to-fix-ssh-issue-on-a-linux-ec2-instance-3inn)</b></i>
***

### <b>How "AWSSupport-ExecuteEC2Rescue" works in the background ?💻</b>

>  - This Automation document executes **[EC2Rescue for Linux](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/Linux-Server-EC2Rescue.html)** on an offline instance (*which does not need to have the SSM agent installed or be user-accessible*) by creating rescue resources, moving the root volume of the target instance to the rescue instance, and later reattaching the root volume to the original instance. - **All  will be done via an automated set of steps in the document, which are as listed here:**

![How AWS Systems Manager Automation Document - "AWSSupport-ExecuteEC2Rescue" works in the background](https://dev-to-uploads.s3.amazonaws.com/i/bt4xxh2mdgclz287c35w.jpg)

To know more  on this document walkthrough, please check **[here.](https://docs.aws.amazon.com/systems-manager/latest/userguide/automation-ec2rescue.html)**

### <b>Prerequisites🔎⚙️</b>
*Before you begin with next steps, you should  have:* 
<ul>
<li><b>Required: "Instance ID"</b> of the unreachable instance. You will specify this ID in the procedure.</li>
<li>In addition, there are some <b>optional parameters</b>, which you can  refer <b><a href="https://docs.aws.amazon.com/systems-manager/latest/userguide/automation-ec2rescue.html#automation-ec2rescue-begin" target="_blank"> here</b></a></li>
<li><b>The IAM role</b> for this execution. If no role is specified, AWS Systems Manager Automation will use <b><u>your IAM permissions of the user logged in</u></b> to execute this document. To know more  on granting permissions by using IAM policies, please refer <b><a href="https://docs.aws.amazon.com/systems-manager/latest/userguide/automation-ec2rescue.html#automation-ec2rescue-access-iam" target="_blank"> here</b></a></li>
</ul>

### <b>I will explain more  in details with below real time example.🎯🎯</b>
Now, I  will show you how to use AWS Systems Manager Automation Document - **["AWSSupport-ExecuteEC2Rescue"](https://console.aws.amazon.com/systems-manager/automation/execute/AWSSupport-ExecuteEC2Rescue)** in real time use-case for a common Linux SSH issue.
![2](https://dev-to-uploads.s3.amazonaws.com/i/5exsj411223mssmnhoou.jpg)

### **Issue😨😨:-**
*(I have changed permissions of **/home directory to 777** and now I am not able to SSH and getting **“Permission denied"**)*🚫😜
![3](https://dev-to-uploads.s3.amazonaws.com/i/m8adwplqooqtberzgsmw.jpg)

<b>Checking SSH verbose output:</b>

    # ssh -vvv -i "eu-west-2_key_pair.pem" ec2-user@ec2-xxxxxxxxxxx.eu-west-2.compute.amazonaws.com

![4](https://dev-to-uploads.s3.amazonaws.com/i/g1nf4c571719grkkv1yv.jpg)

> **Getting permission denied error😖😖**

### <b> Now , let’s use **["AWSSupport-ExecuteEC2Rescue"](https://console.aws.amazon.com/systems-manager/automation/execute/AWSSupport-ExecuteEC2Rescue)** Automation Document to Fix this issue:</b>

•	Open the **AWS Systems Manager** via EC2 Console -- Type [**"Systems Manager"**](https://console.aws.amazon.com/systems-manager/)  
![5](https://dev-to-uploads.s3.amazonaws.com/i/v6vmjigxteh1f6nz2dai.jpg)

•	In the navigation pane, choose **Automation** ----> Choose **Execute Automation**.
![6](https://dev-to-uploads.s3.amazonaws.com/i/os3x09pk0fs9oaisi3lm.jpg)

•	In the Automation document section categories, You can  choose **"Self service support workflows"** and check for the Document named - **"AWSSupport-ExecuteEC2Rescue"** like below and click **“Next”**
![7](https://dev-to-uploads.s3.amazonaws.com/i/mib1v5t84iz047csmwh2.jpg)

•	Please provide the **Instance ID** of **unreachable Instance (Required)** and click **Execute** button to start Automation workflow:
![8](https://dev-to-uploads.s3.amazonaws.com/i/8sohsqwri41dtt9qpiaa.jpg)

***Some Information on Input Parameters:***💭💭

<code>**Mandatory / Required:**</code>

 - **UnreachableInstanceId	: (Required) ID** of your unreachable EC2 instance.
**IMPORTANT:** AWS Systems Manager Automation stops this instance, and creates an AMI before attempting any operations. Data stored in instance store volumes will be lost. The public IP address will change if you are not using an Elastic IP.
 - **EC2RescueInstanceType : (Required)** The EC2 instance type for the EC2Rescue instance. Recommended size: t2.small. *(by default it is
   auto-selected )*

<code>**Optional but could be really useful:**</code>

 - **LogDestination: (Optional)** S3 bucket name in your account where you want to upload the troubleshooting logs. 
   > Make sure the bucket policy does not grant unnecessary read/write
   > permissions to parties that do not need access to the collected logs.

**Now, Let’s proceed further**
•	Once you click on **"Execute"** Button, the Automation will start and you can  the Status **“In Progress”:**
![9](https://dev-to-uploads.s3.amazonaws.com/i/ib9xgphjvu6dzufjpbdh.jpg)

•	To see more  details you can  click on the **"Execution ID"**
![10](https://dev-to-uploads.s3.amazonaws.com/i/4yezbv78tl2etc39oea3.jpg)

### <b>**Checking what it does in the background:**</b>
•	Procedure creates an **SSM helper Instance / recovery**  Instance with name tag **AWS-Support-EC2Rescue-I-xxxx**
![11](https://dev-to-uploads.s3.amazonaws.com/i/bq2qsprm2c8uw4tp0zvj.jpg)

•	Creates an **AMI** as well for Backup Purpose before it executes further steps:
![12](https://dev-to-uploads.s3.amazonaws.com/i/7kbmz0em6w9pexxumt0i.jpg)

•	Stops the **Problematic Unreachable Instance** and Detaches the Root Volume from it and Attaches on the SSM enabled Recovery/Helper Instance:
![13](https://dev-to-uploads.s3.amazonaws.com/i/1k0f2smyn03s6d1f99hg.jpg)

•	Then procedure runs **[EC2Rescue for Linux](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/Linux-Server-EC2Rescue.html)** over helper Instance to Fix the Issue and you can  also track the steps below:
![14](https://dev-to-uploads.s3.amazonaws.com/i/vp4osg04efdg7bvxrj32.jpg)

•	You can  always  check the details of each **"Execution ID"** and its Associated steps using **“Step ID”** , For example, checking details of **“Step-2”** below:
![15](https://dev-to-uploads.s3.amazonaws.com/i/cx4e7ulig9fg32jx1757.jpg)

**Please Note** – 

> “It may show a step failed for Windows since the Instance is Linux,
> So don't worry about  it”🙆‍♂️👍

![16](https://dev-to-uploads.s3.amazonaws.com/i/nv7p3a31rvw9443xuyu2.jpg)

•	You can  monitor the **Overall Status** of the Procedure using the Execution Status tab under Automation Executions and wait for it to get **“Success**” which marks it as complete.
![17](https://dev-to-uploads.s3.amazonaws.com/i/45xz8gn6904a3ay6s588.jpg)

•	*Now, checking the Instance state again - <code>You can  see the Rescue / SSM enabled Helper Instance has been terminated and Automation have Started the Problematic Unreachable Instance again after fixing the issue:</code>*
![18](https://dev-to-uploads.s3.amazonaws.com/i/tkdlj7sshuwk3xwflswc.jpg)

**Now, Let's Try  to connect to the Instance:**👩‍💻🤔
![19](https://dev-to-uploads.s3.amazonaws.com/i/na1yttk0npylnxeeltfe.jpg)

**I am able to SSH and Issue has been fixed now. ✅✅🏁🏁**

### <b>**Interested to know about logs to see what was fixed on Instance to gain access?**</b>❓🙋‍♂️

•	You can  check the details of  Step ID **“getEC2RescueForLinuxResult”** which will provide you a location of the log presence on instance.
![20](https://dev-to-uploads.s3.amazonaws.com/i/fmkeijv6d1mpiatoqcpa.jpg)

•	Please make a note of the location displayed there for example, In my case - The output logs are located in: **/var/tmp/ec2rl/2020-06-25T01_44_08.560197**
![21](https://dev-to-uploads.s3.amazonaws.com/i/35x1yg9rvnmp83sgsdbu.jpg)

•	Now, I will check the **Log location** on instance and see detailed information as follows:
![22](https://dev-to-uploads.s3.amazonaws.com/i/bli4004u8xjjzf1lsxoq.jpg)

>> <b><u>Hence, I have also identified what fixes have been applied by EC2Rescue for Linux over instance to fix the issue in an automated way.</b></u>

### <b>**Conclusion**</b>🎯🧐📚🤖
**[AWSSupport-ExecuteEC2Rescue](https://docs.aws.amazon.com/systems-manager/latest/userguide/automation-awssupport-executeec2rescue.html)**  is a new Automation document that automates all  the steps required to fix common issues on your unreachable Windows & Linux instance using respective **[EC2Rescue for Linux](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/Linux-Server-EC2Rescue.html)** &  **[EC2Rescue for Windows](https://docs.aws.amazon.com/AWSEC2/latest/WindowsGuide/Windows-Server-EC2Rescue.html)** tool tools, which is a framework for executing diagnostic and troubleshooting modules for analyzing and remediating issues.

<code>**Overall**, It will save a lot  of time which is being invested with manual process of detach, attach volume and further steps to troubleshoot issues. This document does all  the steps for you in an automated way in just  few  minutes.</code>

<b>Tip</b>
> With the integration between CloudWatch Events and Systems Manager
> Automation, you can  run AWSSupport-ExecuteEC2Rescue automatically in
> response to an event in your infrastructure.

![SSH issue Fixed](https://media.giphy.com/media/3og0ICG4WxdKSRzE3K/giphy.gif)

***
<i><b>Also, I would like to mention that above I've guided you through Console way to get familiar with the workflow. If you're interested in using AWS CLI, please check out my [Blog Post](https://dev.to/dineshrathee12/via-aws-cli-automating-my-post-using-aws-systems-manager-automation-document-to-fix-ssh-issue-on-a-linux-ec2-instance-3inn)</b></i>
***

<b>Thanks for reading.<b>
Any feedback, please write it to me here in comments..
<b>Also, 🤝🤗You can connect with me🤝🤗</b>
<ul>
<li><b><a href="https://www.linkedin.com/in/%E2%80%8Bdineshrathee12" target="_blank">LinkedIn</a></li>
<li><b><a href="https://dev.to/dineshrathee12" target="_blank">DevTo</a></li>
<li><b><a href="https://github.com/dineshrathee12" target="_blank">GitHub</a></li>
<li><b><a href="https://twitter.com/dinesh_rathee12" target="_blank">Twitter</a></li>
</ul>