Elastic Agent stuck "Pending" on Custom Elastic Image
Asked by Karol S?jko:

Ok so I think I've got everything pretty much running how I want - the only thing is the Agent seems to be stuck when I start an instance.

So here's what I did:

1. I've created an instance based on Atlassian maintained AMI (ami-5be8f232) and customised it with provisioning handled from Vagrant+Puppet. So I ended up with an customised dev environment based on an Atlassian AMI
2. I've created an image out of that so I could reproduce the instances with that
3. Now every time I try to start an instance based on the new image the instance starts perfectly but the agent is stuck "pending".
4. When I \`tail -f /home/bamboo/bamboo-elastic-agent.out\` it seems to be stuck on starting the tunnel server. Here's the output:

> 2014-06-11 10:29:49,620 INFO \[main\] \[S3Utils\] Found 1592 files in /opt/bamboo-elastic-agent
>
> 2014-06-11 10:29:49,620 INFO \[main\] \[S3Utils\] Generating the list of files to fetch from S3...
>
> 2014-06-11 10:29:49,645 INFO \[main\] \[S3Utils\] Generating the list of files to remove...
>
> 2014-06-11 10:29:49,651 INFO \[main\] \[S3Utils\] Removing 1174 files from /opt/bamboo-elastic-agent
>
> 2014-06-11 10:29:51,995 INFO \[main\] \[S3Utils\] Fetching 171 files to /opt/bamboo-elastic-agent
>
> 2014-06-11 10:30:05,991 INFO \[main\] \[S3Utils\] Fetched 135 MB from S3
>
> 2014-06-11 10:30:05,996 INFO \[main\] \[ElasticAgentInstaller\] Starting \[java, -server, -Xms32m, -Xmx256m, -XX:MaxPermSize=128m, -XX:+HeapDumpOnOutOfMemoryError, -Dimagefiles.version=3.3, -Dbamboo.agent.installDir=/opt/bamboo-elastic-agent, -cp, /opt/bamboo-elastic-agent/boot/annotations-13.0.jar:/opt/bamboo-elastic-agent/boot/atlassian-bamboo-core-agent-bootstrap-5.5-OD-04-0004.jar:/opt/bamboo-elastic-agent/boot/gson-2.2.2-atlassian-1.jar:/opt/bamboo-elastic-agent/boot/commons-io-2.4.jar:/opt/bamboo-elastic-agent/boot/atlassian-bamboo-agent-elastic-5.5-OD-04-0004.jar:/opt/bamboo-elastic-agent/boot/jackson-core-2.1.1.jar:/opt/bamboo-elastic-agent/boot/atlassian-tunnel-0.20.jar:/opt/bamboo-elastic-agent/boot/stax-api-1.0-2.jar:/opt/bamboo-elastic-agent/boot/atlassian-bamboo-api-agent-bootstrap-5.5-OD-04-0004.jar:/opt/bamboo-elastic-agent/boot/commons-httpclient-3.0.1.jar:/opt/bamboo-elastic-agent/boot/atlassian-util-concurrent-2.4.1.jar:/opt/bamboo-elastic-agent/boot/joda-time-2.3.jar:/opt/bamboo-elastic-agent/boot/log4j-1.2.15.jar:/opt/bamboo-elastic-agent/boot/commons-codec-1.4.jar:/opt/bamboo-elastic-agent/boot/commons-lang-2.6.jar:/opt/bamboo-elastic-agent/boot/atlassian-bamboo-agent-bootstrap-5.5-OD-04-0004.jar:/opt/bamboo-elastic-agent/boot/jackson-databind-2.1.1.jar:/opt/bamboo-elastic-agent/boot/fugue-1.1.jar:/opt/bamboo-elastic-agent/boot/guava-10.0.1.jar:/opt/bamboo-elastic-agent/boot/aws-java-sdk-1.7.1.jar:/opt/bamboo-elastic-agent/boot/httpclient-4.2.5.jar:/opt/bamboo-elastic-agent/boot/commons-logging-1.0.4.jar:/opt/bamboo-elastic-agent/boot/atlassian-aws-1.0.59.jar:/opt/bamboo-elastic-agent/boot/jackson-annotations-2.1.1.jar:/opt/bamboo-elastic-agent/boot/bcprov-jdk15on-1.48.jar:/opt/bamboo-elastic-agent/boot/bcpkix-jdk15on-1.48.jar:/opt/bamboo-elastic-agent/boot/atlassian-annotations-0.4.jar:/opt/bamboo-elastic-agent/boot/jsr305-1.3.9.jar:/opt/bamboo-elastic-agent/boot/atlassian-bamboo-agent-elastic-shared-5.5-OD-04-0004.jar:/opt/bamboo-elastic-agent/boot/httpcore-4.2.5.jar:, com.atlassian.bamboo.agent.elastic.client.ElasticAgentBootstrap\]
>
> 2014-06-11 10:30:06,209 INFO \[main\] \[ElasticAgentBootstrap\] Starting Agent Bootstrap using Java 1.6.0_45 from Sun Microsystems Inc.
>
> 2014-06-11 10:30:06,446 INFO \[main\] \[ElasticAgentBootstrap\] Using tunnelling. Registering 'httpt' and 'httpst' protocols.
>
> 2014-06-11 10:30:06,471 INFO \[main\] \[ElasticAgentBootstrap\] Starting tunnel server
>
> 2014-06-11 11:00:06,472 ERROR \[main\] \[ElasticAgentBootstrap\] Fatal exception.
>
> java.util.concurrent.TimeoutException: The server has failed to contact the agent within 30 minutes. Assuming that the server is unreachable and terminating the agent.
>
> at com.atlassian.bamboo.agent.elastic.client.ElasticAgentBootstrap.startTunnelServer(ElasticAgentBootstrap.java:159)
>
> at com.atlassian.bamboo.agent.elastic.client.ElasticAgentBootstrap.run(ElasticAgentBootstrap.java:88)
>
> at com.atlassian.bamboo.agent.elastic.client.ElasticAgentBootstrap.main(ElasticAgentBootstrap.java:56)


Answers:
author: Karol S?jko
body: yeah I was able to ssh to it via Vagrant tool-set. But Bamboo wasn't able to access it. In the end I have prepared the environment manually based on an Atlassian AMI and now it's accessible.


author: Przemek Bruski
body: Is the tunnel port (26224) on your EC2 instance reachable from outside?


author: Fabian Knorr
body: Hi Karol,

this looks like a network error. Are you running the Elastic Agents in a VPC? If so, are routing-tables and firewall correctly configured? I mean both, at AWS and at you own LAN.

comments: author: Karol S?jko
body: No, the Elastic Agents are not run on a VPC


Tags: cloud

elastic-agent