bamboo elastic agent permission denied on S3 sync
Asked by Jeff Behl:

Seen a number of posts about this sort of issue...don't get why it seems to happen regularly. Atlassian - looks like something you need to fix? from us-west-1

```
[root@ip-10-35-12-70 bamboo]# /opt/bamboo-elastic-agent/bin/bamboo-elastic-agentSyncing Elastic Bamboo Agent files...2015-12-13 04:30:22,812 INFO [main] [S3Sync] Syncing from: bamboo-agent-release-us-w1/5.9.7/b3f798e03f020d72f10564280b47840ba203ae32/ to /opt/bamboo-elastic-agent2015-12-13 04:30:24,521 INFO [main] [S3Synchroniser] Syncing s3://bamboo-agent-release-us-w1/5.9.7/b3f798e03f020d72f10564280b47840ba203ae32/ to /opt/bamboo-elastic-agent2015-12-13 04:30:24,536 INFO [main] [AmazonClients] Detecting bucket location for [bamboo-agent-release-us-w1]2015-12-13 04:30:25,812 WARN [main] [AmazonClients] Unable to get bucket location for [bamboo-agent-release-us-w1], using default. Error: com.amazonaws.services.s3.model.AmazonS3Exception: Access Denied (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: 0551DDC8AC66E08A), S3 Extended Request ID: vufOZs1Jw7DKEKq2bwAT6bnE/ZUPQFdI+7jtK7ITL9jLTCuV9GhBZwUXvPa6Q8TCLrSUvhc+7lc=2015-12-13 04:30:25,812 INFO [main] [S3Synchroniser] Fetching the list of remote objects...Exception in thread "main" com.amazonaws.services.s3.model.AmazonS3Exception: Access Denied (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: B8E6EC6AF92964D7), S3 Extended Request ID: sic01W4ilb9JpvxXGb/hpsM6oBtqNjc7fmRwCildB9PLnKgMYHJvOvwviiuyySv0Pxo0E8+K0fA=	at com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:1182)	at com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:770)	at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:489)	at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:310)	at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3604)	at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3557)	at com.amazonaws.services.s3.AmazonS3Client.listObjects(AmazonS3Client.java:647)	at com.amazonaws.services.s3.AmazonS3Client.listObjects(AmazonS3Client.java:632)	at com.atlassian.aws.s3.S3Synchroniser.getObjectNamesAndHashes(S3Synchroniser.java:341)	at com.atlassian.aws.s3.S3Synchroniser.sync(S3Synchroniser.java:163)	at com.atlassian.bamboo.agent.elastic.S3Sync.sync(S3Sync.java:72)	at com.atlassian.bamboo.agent.elastic.installer.ElasticAgentInstaller.install(ElasticAgentInstaller.java:76)	at com.atlassian.bamboo.agent.elastic.installer.ElasticAgentInstaller.main(ElasticAgentInstaller.java:173)
```


Answers:
author: Jeff Behl
body: So here's the answer: Atlassian's S3 bucket that the elastic agent uses to sync some jars and whatnot is restricted to EC2 IP addresses. Our VPC traffic runs through a VPN link and exits through our data center, meaning the source appeared to be non-EC2. The workaround was to enable EC2 VPN endpoints which allows for internal VPC hosts to access S3 directly:

<http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/vpc-endpoints.html>


author: Jeff Behl
body: Apologies with the conversation with myself, but a few more data points...

The problem seems to reside round the detection of the region, perhaps due to the custom image running from inside of a VPC? Running a stock Ubuntu image loads fine:

```
2015-12-13 20:44:47,248 INFO [main] [S3Sync] Syncing from: bamboo-agent-release-us-w1/5.9.7/b3f798e03f020d72f10564280b47840ba203ae32/ to /opt/bamboo-elastic-agent2015-12-13 20:44:51,092 INFO [main] [S3Utils] Syncing s3://bamboo-agent-release-us-w1/5.9.7/b3f798e03f020d72f10564280b47840ba203ae32/ to /opt/bamboo-elastic-agent2015-12-13 20:44:51,092 INFO [main] [AmazonClients] Detecting bucket location for [bamboo-agent-release-us-w1]2015-12-13 20:44:53,079 INFO [main] [AmazonClients] Set S3 endpoint to: s3-us-west-1.amazonaws.com2015-12-13 20:44:53,079 INFO [main] [S3Utils] Fetching the list of remote objects...
```

It should be noted it is running outside of a VPC and has a public address.

The custom image I've created and started via bamboo resides inside of a VPC and does not have a public IP address. It is able to talk to the internet, though through a VPN connection. It difference seems to be:

```
2015-12-13 04:30:25,812 WARN [main] [AmazonClients] Unable to get bucket location for [bamboo-agent-release-us-w1], using default.
```

Any way I can override the S3 endpoint? Or influence the "detecting bucket location" ?


author: Jeff Behl
body: @Przemyslaw Bruski - here's the full (just tried). And I should have mentioned this earlier: this is a custom image that I've created to do this. The instance was started by bamboo, however, but sits there at the "Pending" state from the Bamboo UI. The below is after I've logged into this instance to see what could have gone wrong and tried running the bamboo-elastic-agent manually. It's also in the "/home/bamboo/bamboo-elastic-agent.out" file, so I'm assuming it's running correctly, just getting that error and failing..

```
[root@ip-10-35-12-72 bin]# ./bamboo-elastic-agentSyncing Elastic Bamboo Agent files...2015-12-13 19:23:04,364 INFO [main] [S3Sync] Syncing from: bamboo-agent-release-us-w1/5.9.7/b3f798e03f020d72f10564280b47840ba203ae32/ to /opt/bamboo-elastic-agent2015-12-13 19:23:05,885 INFO [main] [S3Synchroniser] Syncing s3://bamboo-agent-release-us-w1/5.9.7/b3f798e03f020d72f10564280b47840ba203ae32/ to /opt/bamboo-elastic-agent2015-12-13 19:23:05,886 INFO [main] [AmazonClients] Detecting bucket location for [bamboo-agent-release-us-w1]2015-12-13 19:23:07,125 WARN [main] [AmazonClients] Unable to get bucket location for [bamboo-agent-release-us-w1], using default. Error: com.amazonaws.services.s3.model.AmazonS3Exception: Access Denied (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: 1061719D3D6F6E92), S3 Extended Request ID: 1V4qvBwT0neOf+Xtd7avXigpx5XkvO4et33OVQtCEzLpcJFjOQB8+YfNXoLjM1uAtEl5in8yHII=2015-12-13 19:23:07,125 INFO [main] [S3Synchroniser] Fetching the list of remote objects...Exception in thread "main" com.amazonaws.services.s3.model.AmazonS3Exception: Access Denied (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: 5CA1ECA6CA63A5FF), S3 Extended Request ID: 00pQkCl5ITT5AGCqrQ1krQInpbXh4ofnc7R5XWxoWk6bR3MXAwoYK4cRhdnW/rZ25+GiatrvcO0=	at com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:1182)....
```


author: Przemek Bruski
body: Is it still happening?

comments: author: Jeff Behl
body: yep...just tried:2015-12-13 19:23:07,125 WARN \[main\] \[AmazonClients\] Unable to get bucket location for \[bamboo-agent-release-us-w1\], using default. Error: com.amazonaws.services.s3.model.AmazonS3Exception: Access Denied (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: 1061719D3D6F6E92), S3 Extended Request ID: 1V4qvBwT0neOf+Xtd7avXigpx5XkvO4et33OVQtCEzLpcJFjOQB8+YfNXoLjM1uAtEl5in8yHII=2015-12-13 19:23:07,125 INFO \[main\] \[S3Synchroniser\] Fetching the list of remote objects...Exception in thread "main" com.amazonaws.services.s3.model.AmazonS3Exception: Access Denied (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: 5CA1ECA6CA63A5FF), S3 Extended Request ID: 00pQkCl5ITT5AGCqrQ1krQInpbXh4ofnc7R5XWxoWk6bR3MXAwoYK4cRhdnW/rZ25+GiatrvcO0=


Tags: amazon

aws

bamboo-administration