New Elastic Bamboo agent stuck pending - AmazonS3Exception in log
Asked by Dentally Portal:

Hi Team,

I've just setup a new custom elastic windows image on amazon EC2.

The connections are working ok and the image has been loaded onto a server, but when I start the agent from within bamboo, the elastic agent gets stuck on "pending".

I've logged onto the instance to take a look at the bamboo-elastic-agent.out log file, and can see the following error. Do I need to open an additional port?

```
JAVA_HOME: C:\Program Files\Java\jdk1.7.0_40 PATH: C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;C:\Windows\System32\WindowsPowerShell\v1.0\;C:\Program Files\Amazon\cfn-bootstrap\;C:\Program Files\Microsoft SQL Server\110\Tools\Binn\;C:\Program Files\Microsoft\Web Platform Installer\;C:\Program Files (x86)\Git\cmd HOME: \Users\Bamboo java version "1.7.0_40"Java(TM) SE Runtime Environment (build 1.7.0_40-b43)Java HotSpot(TM) 64-Bit Server VM (build 24.0-b56, mixed mode)2013-09-19 17:13:56,773 INFO [main] [S3Sync] Syncing from: bamboo-agent-release-us-w2/5.0.2-OD-2/a564255d62b4cb45e6b8e2acebe7358660f32b30/ to /opt/bamboo-elastic-agent2013-09-19 17:14:02,217 INFO [main] [S3Utils] Syncing s3://bamboo-agent-release-us-w2/5.0.2-OD-2/a564255d62b4cb45e6b8e2acebe7358660f32b30/ to \opt\bamboo-elastic-agent2013-09-19 17:14:02,217 INFO [main] [S3Utils] Fetching the list of remote objects...Exception in thread "main" AmazonS3Exception: Status Code: 403, AWS Service: Amazon S3, AWS Request ID: 0CB2B4F164F533B0, AWS Error Code: AccessDenied, AWS Error Message: Access Denied, S3 Extended Request ID: wKkYmQcytOOmP2p5hQC7uiXeNBDZsUPdbFa2VAH3GXUg/+AOxPkpIppcQxSgNukX	at com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:644)	at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:338)	at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:190)	at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:2974)	at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:2945)	at com.amazonaws.services.s3.AmazonS3Client.listObjects(AmazonS3Client.java:478)	at com.amazonaws.services.s3.AmazonS3Client.listObjects(AmazonS3Client.java:462)	at com.atlassian.aws.s3.S3Utils.getObjectNamesAndHashes(S3Utils.java:331)	at com.atlassian.aws.s3.S3Utils.sync(S3Utils.java:159)	at com.atlassian.bamboo.agent.elastic.S3Sync.sync(S3Sync.java:72)	at com.atlassian.bamboo.agent.elastic.installer.ElasticAgentInstaller.main(ElasticAgentInstaller.java:66)
```


Answers:
author: Richard Barnett
body: This has just started to happen for us.

Do you need to do another refresh?


author: Wayne Allan
body: I'm getting the same error message.

bamboo-elastic-agent.log

```
JAVA_HOME: C:\Program Files\Java\jdk1.7.0_40 PATH: C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;C:\Program Files\Amazon\cfn-bootstrap\;C:\Program Files (x86)\Git\cmd;C:\Program Files\Microsoft\Web Platform Installer\;C:\Program Files (x86)\Microsoft ASP.NET\ASP.NET Web Pages\v1.0\;C:\Program Files\Microsoft SQL Server\110\Tools\Binn\;C:\Program Files\NuGet;%JAVA_HOME\bin% HOME: \Users\Bamboo java version "1.7.0_40"Java(TM) SE Runtime Environment (build 1.7.0_40-b43)Java HotSpot(TM) 64-Bit Server VM (build 24.0-b56, mixed mode)2013-09-30 00:44:47,767 INFO [main] [S3Sync] Syncing from: bamboo-agent-release-ap-se2/5.2-OD-2/fa5fd0da044fad69f7648355dce5c5dc8f613e65/ to /opt/bamboo-elastic-agent2013-09-30 00:44:50,903 INFO [main] [S3Utils] Syncing s3://bamboo-agent-release-ap-se2/5.2-OD-2/fa5fd0da044fad69f7648355dce5c5dc8f613e65/ to \opt\bamboo-elastic-agent2013-09-30 00:44:50,903 INFO [main] [S3Utils] Fetching the list of remote objects...Exception in thread "main" AmazonS3Exception: Status Code: 403, AWS Service: Amazon S3, AWS Request ID: F61A967F320E768F, AWS Error Code: AccessDenied, AWS Error Message: Access Denied, S3 Extended Request ID: u2Tq0QiuNyYayZKX5ncnnNLMj7xU4j3XnlVN9kNVNu3lpk/pjtbn+u1ptVvqXy51	at com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:644)	at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:338)	at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:190)	at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:2974)	at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:2945)	at com.amazonaws.services.s3.AmazonS3Client.listObjects(AmazonS3Client.java:478)	at com.amazonaws.services.s3.AmazonS3Client.listObjects(AmazonS3Client.java:462)	at com.atlassian.aws.s3.S3Utils.getObjectNamesAndHashes(S3Utils.java:331)	at com.atlassian.aws.s3.S3Utils.sync(S3Utils.java:159)	at com.atlassian.bamboo.agent.elastic.S3Sync.sync(S3Sync.java:72)	at com.atlassian.bamboo.agent.elastic.installer.ElasticAgentInstaller.main(ElasticAgentInstaller.java:66)
```

I raised a bug for it <https://jira.atlassian.com/browse/BAM-13826>

I'm only guessing but is it that the automated deployments of the new agents aren't setting the right permissions on the s3 buckets?


author: Dentally Portal
body: Thanks - I'm now getting an error from within Bamboo on Demand itself when starting the instance - so I can't yet determine if your change has fixed my issue.

I'm getting "Failed to start elastic instances: Error when starting a new instance".

The System errors page in Bamboo contains no errors. Would this be in the log? How do I get at the Bamboo log using On Demand?

comments: author: Przemek Bruski
body: Please contact support, they'll get the log and probably get back to devs to improve that error message.


author: Thivagar N
body: "failed to start elastic instances error when starting a new instance"

Am getting this error.. please help me


author: Przemek Bruski
body: I've refreshed the bucket policies, can you try again?


Tags: bamboo-administration