Elastic Images Terminate
Asked by Galen Pyle:

Hi, I'm trying to create custom build server instances from Bamboo elastic images. I'm installing .NET 4.5 and a Visual Studio Shell on the drive. But the elastic images are terminating suddenly. I'll be logged on and moving forward with building the build server, and it just shuts down and terminates. Please see the below log from Bamboo. Why are the Bamboo elastic images terminating unexpectedly??

|-------------------------------------------------------------------------------------------------------------------------|
| Oct 12, 2012 1:53:10 PM Requested that new elastic instance be created for configuration: Windows .NET 4.5 Build Server |
| Oct 12, 2012 1:53:10 PM Elastic instance \[i-80f9dbfd\] transitioned from STARTING to IDENTIFIED.                       |
| Oct 12, 2012 1:53:52 PM Elastic instance \[i-80f9dbfd\] transitioned from IDENTIFIED to RUNNING.                        |
| Oct 12, 2012 2:02:35 PM An elastic agent is loading on instance: \[i-80f9dbfd\]                                         |
| Oct 12, 2012 2:04:01 PM Elastic Agent "Elastic Agent on i-80f9dbfd" started on instance i-80f9dbfd                      |
| Oct 12, 2012 3:04:12 PM An elastic agent is loading on instance: \[i-80f9dbfd\]                                         |
| Oct 12, 2012 3:09:17 PM Elastic Agent "Elastic Agent on i-80f9dbfd" stopped on instance i-80f9dbfd                      |
| Oct 12, 2012 3:09:51 PM Elastic Agent "Elastic Agent on i-80f9dbfd (2)" started on instance i-80f9dbfd                  |
| Oct 12, 2012 3:14:36 PM Elastic instance \[i-80f9dbfd\] transitioned from RUNNING to SHUTTING_DOWN.                     |
| Oct 12, 2012 3:14:57 PM Elastic instance \[i-80f9dbfd\] transitioned from SHUTTING_DOWN to TERMINATED.                  |
| Oct 12, 2012 3:14:57 PM Detected that the elastic instance \[i-80f9dbfd\] has been terminated.                          |
| Oct 12, 2012 3:14:57 PM Elastic Agent "Elastic Agent on i-80f9dbfd (2)" stopped on instance i-80f9dbfd                  |


Answers:
author: Galen Pyle
body: I've spoken with Atlassian technical support and the problem is caused by the automatic elastic instance management configuration setting. It is the best practice to disable this feature. There is a Bamboo agent that runs on each Bamboo build server instance. If this agent stops and the automatic elastic instance management configuration setting is enabled, then the server auto-terminates. The agent may stop if you restart the Bamboo build server instance.

comments: author: Jeff Hansen
body: I have disabled everything that could possibly auto-terminate my elastic images, but they are still terminating on their own (like once a month - fairly infrequently but still annoying). Any other ideas to ensure that the image doesn't shut down? Is Atlasssian shutting down their EC2 cloud group or something like that periodically?


author: derek williams
body: Hi,

I didn't see a specific answer here, it's an old thread so, I suspect it was resolved. We recently encountered the same symptoms, just leaving my experience for future visitors.

Bamboo would launch elastic instance, the instance would terminate about 3-4 minutes later, the agent never moved out of pending.

Cause:

* Our AMI was setup to create 2 volumes when new instances are launched.

* When the instance was terminated, the second volume was being left behind. I suspect this has something to do with our AMI profile permissions. Eventually, AWS wouldn't provision anymore volumes because, we had too many created. (got up to around 100 orphaned volumes before we noticed)

Result:

* Bamboo would attempt to create new elastic instances

* AWS would start the instance, then terminate it because the volume couldn't be created

* Inside Bamboo it appeared the agent was just randomly going offline

<br />

For us, we just adjust the AMI to only create 1 volume and used the asw console to cleanup the volumes.

cheers

comments: author: 140401 kindia
body: Hi ,

I am also observing similar error. Bamboo elastic agents get terminated while building plans.

any workaround and solution?


author: Przemek Bruski
body: I suspect that the agent is having problems starting up, shuts down, gets restarted (that why you see the agent marked with (2) ). Eventually Bamboo decides the agent won't come up and kills the instance.

Are you stopping/starting/restarting the instance in the meantime?

If not, what does the agent log say (you can find it in the agent home directory)?

comments: author: Galen Pyle
body: Thanks for your response! Please see the attached Elastic Agent Usage History. Each of the Windows Bamboo build server instances have auto-terminated. I created the three most recent elastic agents as a test. I never remoted into these three instances, but they each terminated at 7:48 PM on Oct 14. I do not stop/start the instances after their initial creation. I did restart some of these build servers after I installed .NET 4.5. We use AWS on a daily basis and we only have this problem with the Bamboo elastic agents. This is a serious impediment to our progress. Thanks so much for your time \& consideration.


author: Galen Pyle
body: **[Elastic Agent History](https://mcsproject.atlassian.net/builds/admin/elastic/viewElasticAgentHistory.action "View Elastic Agent History") ? Elastic Agent on i-26f4d95b (elastic)**

Elastic agent on instance i-26f4d95b, configuration: Windows .NET 4.5 Build Server

You can view historical information for this elastic agent in this page. This agent is offline and will never come back alive.

**Agent Details**

0%0 / 0 Successful

Elastic instance**Instance i-26f4d95b** ([Windows .NET 4.5 Build Server](https://mcsproject.atlassian.net/builds/admin/elastic/image/configuration/viewElasticImageConfiguration.action?configurationId=983041) image configuration)

The elastic instance this agent is/was running on.

Last startup timeOct 12, 2012 4:06:53 PM

Last shutdown timeOct 14, 2012 7:48:07 PM

Up time2 days, 3 hours, 41 minutes

**Build History**

The agent was building for **0%** of the up time.

There are no builds to display.

**[Elastic Agent History](https://mcsproject.atlassian.net/builds/admin/elastic/viewElasticAgentHistory.action "View Elastic Agent History") ? Elastic Agent on i-80f9dbfd (elastic)**

Elastic agent on instance i-80f9dbfd, configuration: Windows .NET 4.5 Build Server

You can view historical information for this elastic agent in this page. This agent is offline and will never come back alive.

**Agent Details**

0%0 / 0 Successful

Elastic instance**Instance i-80f9dbfd** ([Windows .NET 4.5 Build Server](https://mcsproject.atlassian.net/builds/admin/elastic/image/configuration/viewElasticImageConfiguration.action?configurationId=983041) image configuration)

The elastic instance this agent is/was running on.

Last startup timeOct 12, 2012 2:04:01 PM

Last shutdown timeOct 12, 2012 3:09:17 PM

Up time1 hour, 5 minutes

**Build History**

The agent was building for **0%** of the up time.

There are no builds to display.


author: Galen Pyle
body: **Elastic Agent Usage History**

Listed below are all elastic agents that have run in the past. This does not include any currently running elastic agents, you can view these on the [Elastic Instances page](https://mcsproject.atlassian.net/builds/admin/elastic/manageElasticInstances.action)

|                                                       **Agent**                                                        |                                                                             **Configuration**                                                                              |      **Start Time**      |    **Shutdown Time**     |         **Uptime**          | **Successful Jobs** |                                                                                                                                     **Operations**                                                                                                                                      |
|------------------------------------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------|--------------------------|--------------------------|-----------------------------|---------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| [Elastic Agent on i-24f4d959](https://mcsproject.atlassian.net/builds/admin/agent/viewAgent.action?agentId=851978)     | [Windows .NET 4.5 Build Server](https://mcsproject.atlassian.net/builds/admin/elastic/image/configuration/viewElasticImageConfiguration.action?configurationId=983041)     | 12 Oct 2012, 4:09:16 PM  | 14 Oct 2012, 7:48:07 PM  | 2 days, 3 hours, 38 minutes | No builds           | [View](https://mcsproject.atlassian.net/builds/admin/agent/viewAgent.action?agentId=851978) \| [Delete](https://mcsproject.atlassian.net/builds/admin/agent/removeAgent.action?agentId=851978&returnUrl=/admin/elastic/viewElasticAgentHistory.action "permanently remove this agent.") |
| [Elastic Agent on i-26f4d95b](https://mcsproject.atlassian.net/builds/admin/agent/viewAgent.action?agentId=851977)     | [Windows .NET 4.5 Build Server](https://mcsproject.atlassian.net/builds/admin/elastic/image/configuration/viewElasticImageConfiguration.action?configurationId=983041)     | 12 Oct 2012, 4:06:53 PM  | 14 Oct 2012, 7:48:07 PM  | 2 days, 3 hours, 41 minutes | No builds           | [View](https://mcsproject.atlassian.net/builds/admin/agent/viewAgent.action?agentId=851977) \| [Delete](https://mcsproject.atlassian.net/builds/admin/agent/removeAgent.action?agentId=851977&returnUrl=/admin/elastic/viewElasticAgentHistory.action "permanently remove this agent.") |
| [Elastic Agent on i-42f4d93f](https://mcsproject.atlassian.net/builds/admin/agent/viewAgent.action?agentId=851976)     | [Windows .NET 4.5 Build Server](https://mcsproject.atlassian.net/builds/admin/elastic/image/configuration/viewElasticImageConfiguration.action?configurationId=983041)     | 12 Oct 2012, 4:06:39 PM  | 14 Oct 2012, 7:48:07 PM  | 2 days, 3 hours, 41 minutes | No builds           | [View](https://mcsproject.atlassian.net/builds/admin/agent/viewAgent.action?agentId=851976) \| [Delete](https://mcsproject.atlassian.net/builds/admin/agent/removeAgent.action?agentId=851976&returnUrl=/admin/elastic/viewElasticAgentHistory.action "permanently remove this agent.") |
| [Elastic Agent on i-80f9dbfd (2)](https://mcsproject.atlassian.net/builds/admin/agent/viewAgent.action?agentId=851975) | [Windows .NET 4.5 Build Server](https://mcsproject.atlassian.net/builds/admin/elastic/image/configuration/viewElasticImageConfiguration.action?configurationId=983041)     | 12 Oct 2012, 3:09:51 PM  | 12 Oct 2012, 3:14:57 PM  | 5 minutes                   | 0 / 1               | [View](https://mcsproject.atlassian.net/builds/admin/agent/viewAgent.action?agentId=851975) \| [Delete](https://mcsproject.atlassian.net/builds/admin/agent/removeAgent.action?agentId=851975&returnUrl=/admin/elastic/viewElasticAgentHistory.action "permanently remove this agent.") |
| [Elastic Agent on i-80f9dbfd](https://mcsproject.atlassian.net/builds/admin/agent/viewAgent.action?agentId=851974)     | [Windows .NET 4.5 Build Server](https://mcsproject.atlassian.net/builds/admin/elastic/image/configuration/viewElasticImageConfiguration.action?configurationId=983041)     | 12 Oct 2012, 2:04:01 PM  | 12 Oct 2012, 3:09:17 PM  | 1 hour, 5 minutes           | No builds           | [View](https://mcsproject.atlassian.net/builds/admin/agent/viewAgent.action?agentId=851974) \| [Delete](https://mcsproject.atlassian.net/builds/admin/agent/removeAgent.action?agentId=851974&returnUrl=/admin/elastic/viewElasticAgentHistory.action "permanently remove this agent.") |
| [Elastic Agent on i-84496bf9 (2)](https://mcsproject.atlassian.net/builds/admin/agent/viewAgent.action?agentId=851973) | [Windows .NET 4.5 Build Server](https://mcsproject.atlassian.net/builds/admin/elastic/image/configuration/viewElasticImageConfiguration.action?configurationId=983041)     | 12 Oct 2012, 1:31:58 PM  | 12 Oct 2012, 1:45:48 PM  | 13 minutes                  | 0 / 1               | [View](https://mcsproject.atlassian.net/builds/admin/agent/viewAgent.action?agentId=851973) \| [Delete](https://mcsproject.atlassian.net/builds/admin/agent/removeAgent.action?agentId=851973&returnUrl=/admin/elastic/viewElasticAgentHistory.action "permanently remove this agent.") |
| [Elastic Agent on i-84496bf9](https://mcsproject.atlassian.net/builds/admin/agent/viewAgent.action?agentId=851972)     | [Windows .NET 4.5 Build Server](https://mcsproject.atlassian.net/builds/admin/elastic/image/configuration/viewElasticImageConfiguration.action?configurationId=983041)     | 12 Oct 2012, 1:12:49 PM  | 12 Oct 2012, 1:40:17 PM  | 27 minutes                  | 0 / 1               | [View](https://mcsproject.atlassian.net/builds/admin/agent/viewAgent.action?agentId=851972) \| [Delete](https://mcsproject.atlassian.net/builds/admin/agent/removeAgent.action?agentId=851972&returnUrl=/admin/elastic/viewElasticAgentHistory.action "permanently remove this agent.") |
| [Elastic Agent on i-aa652fd7](https://mcsproject.atlassian.net/builds/admin/agent/viewAgent.action?agentId=851971)     | [Default Image EBS x86_64 (windows)](https://mcsproject.atlassian.net/builds/admin/elastic/image/configuration/viewElasticImageConfiguration.action?configurationId=98308) | 12 Oct 2012, 12:38:13 PM | 12 Oct 2012, 12:51:46 PM | 13 minutes                  | No builds           | [View](https://mcsproject.atlassian.net/builds/admin/agent/viewAgent.action?agentId=851971) \| [Delete](https://mcsproject.atlassian.net/builds/admin/agent/removeAgent.action?agentId=851971&returnUrl=/admin/elastic/viewElasticAgentHistory.action "permanently remove this agent.") |
| [Windows .NET 4.5 Build Server](https://mcsproject.atlassian.net/builds/admin/agent/viewAgent.action?agentId=851970)   | [Default Image EBS x86_64 (windows)](https://mcsproject.atlassian.net/builds/admin/elastic/image/configuration/viewElasticImageConfiguration.action?configurationId=98308) | 10 Oct 2012, 11:17:28 AM | 12 Oct 2012, 12:46:17 PM | 2 days, 1 hour, 28 minutes  | 8 / 28              | [View](https://mcsproject.atlassian.net/builds/admin/agent/viewAgent.action?agentId=851970) \| [Delete](https://mcsproject.atlassian.net/builds/admin/agent/removeAgent.action?agentId=851970&returnUrl=/admin/elastic/viewElasticAgentHistory.action "permanently remove this agent.") |


author: Galen Pyle
body: **[Elastic Agent History](https://mcsproject.atlassian.net/builds/admin/elastic/viewElasticAgentHistory.action "View Elastic Agent History") ? Elastic Agent on i-26f4d95b (elastic)**

Elastic agent on instance i-26f4d95b, configuration: Windows .NET 4.5 Build Server

You can view historical information for this elastic agent in this page. This agent is offline and will never come back alive.

**Agent Details**

0%0 / 0 Successful

Elastic instance**Instance i-26f4d95b** ([Windows .NET 4.5 Build Server](https://mcsproject.atlassian.net/builds/admin/elastic/image/configuration/viewElasticImageConfiguration.action?configurationId=983041) image configuration)

The elastic instance this agent is/was running on.

Last startup timeOct 12, 2012 4:06:53 PM

Last shutdown timeOct 14, 2012 7:48:07 PM

Up time2 days, 3 hours, 41 minutes

**Build History**

The agent was building for **0%** of the up time.

There are no builds to display.

**[Elastic Agent History](https://mcsproject.atlassian.net/builds/admin/elastic/viewElasticAgentHistory.action "View Elastic Agent History") ? Elastic Agent on i-80f9dbfd (elastic)**

Elastic agent on instance i-80f9dbfd, configuration: Windows .NET 4.5 Build Server

You can view historical information for this elastic agent in this page. This agent is offline and will never come back alive.

**Agent Details**

0%0 / 0 Successful

Elastic instance**Instance i-80f9dbfd** ([Windows .NET 4.5 Build Server](https://mcsproject.atlassian.net/builds/admin/elastic/image/configuration/viewElasticImageConfiguration.action?configurationId=983041) image configuration)

The elastic instance this agent is/was running on.

Last startup timeOct 12, 2012 2:04:01 PM

Last shutdown timeOct 12, 2012 3:09:17 PM

Up time1 hour, 5 minutes

**Build History**

The agent was building for **0%** of the up time.

There are no builds to display.


author: Przemek Bruski
body: Do you have Elastic Instance Management enabled in your Elastic Bamboo configuration?


author: Galen Pyle
body: Elastic Instance Management is currently DISABLED in my Elastic Bamboo configuration. Could this be the cause of the problem? What is the optimal elastic management configuration?

Thanks for your reply!


author: Galen Pyle
body: Elastic Bamboo is configured with our Amazon Web Services Account. Automatic Elastic Instance Management is currently DISABLED in my Elastic Bamboo configuration. Could this be the cause of the problem? What is the optimal elastic management configuration?

Thanks for your reply!

-----------------------------------------------------------------------------------------  

Spot Instances Configuration {#toc-hId--1330391601}
---------------------------------------------------

Support for spot instances is disabled.  

Elastic Bamboo AWS Settings {#toc-hId--442887920}
-------------------------------------------------

AWS account identifier files are not automatically uploaded to new elastic instances.  

Automatic Elastic Instance Management {#toc-hId-444615761}
----------------------------------------------------------

EC2 instances must be *manually* created and terminated as needed.


Tags: aws

cloud

elastic