Exclude files from upload to s3 bucket using "Amazon s3 object task"
Asked by bugretvup:

Hi,

I'm trying to replace "aws s3 sync" script task with "Amazon S3 Object" task, on a server install of bamboo (v6.8.1), in a build plan that uploads content to a s3 bucket.

Trouble i'm facing right now are two:

1) i need to exclude some files from the upload process.

I've tried to get it working using Ant exclude patterns (checking the option "Use Ant patterns to select files"), but with no positive outcome until now.

2) if problem (1) get's solved, and if possible, i would also like to have a 'sync' approach, that is, that the execution of the plan, also removes objects on s3 bucket if they're not present in the local folder anymore.

Documentation i've tried following:

- <https://utoolity.atlassian.net/wiki/spaces/TAWS/pages/19464196/Using+the+Amazon+S3+Object+task+in+Bamboo>

- <http://ant.apache.org/manual/Tasks/defaultexcludes.html>

HAs anyone got this to work?

Thanks

-wm


Answers:
author: Steffen Opel _Utoolity_
body: Hi Werner,  

Thank you for using our apps, and sorry for the trouble. I'm afraid your requirements are simply not supported at this point:

1. The 'Use Ant patterns to select files' option currently only supports *inclusion* of files based on the pattern syntax itself, but not the explicit *exclusion* of files via any of the 'exclude' XML tags.

* As a potential workaround, you might want to consider assembling the desired fileset in a new directory, which would allow to use resp. OS level tooling on copy to exclude parts of the source directory hierarchy (specifics depend on the OS and shell in question).
* I realize that the option's description and linked documentation from Ant are misleading in this regard and we will try to change it so that future users are less likely to run into this, thanks for pointing out this usability flaw.

2. The [Amazon S3 Object task](https://utoolity.atlassian.net/wiki/spaces/TAWS/pages/19464196/Using+the+Amazon+S3+Object+task+in+Bamboo) does indeed not yet support an equivalent for the [aws . s3 . sync](https://docs.aws.amazon.com/cli/latest/reference/s3/sync.html) action, and we still recommend falling back to the [AWS CLI as a workaround for such scenarios](https://utoolity.atlassian.net/wiki/spaces/UAAKB/pages/43515961/How+to+provide+temporary+AWS+credentials+to+the+AWS+Command+Line+Interface+AWS+CLI+via+Bamboo+variables) accordingly. Please watch and vote for the following story to guide our roadmap:

* [As a user, I want a task to sync files with S3 so that upload/download bandwith/time is minimized (UAA-288)](https://utoolity.atlassian.net/browse/UAA-288)

I hope this helps despite lacking an actual solution.  

Cheers,  
Steffen

comments: author: bugretvup
body: Hi Steffen

Thank you for your clarifying answer.

I think that for the time being i'll go back to a shell/system 'aws s3 sync' for this task, at least until UAA-288 is a reality.

Have a nice week

-wm


author: bugretvup
body: Hi CCM,

Sorry, you're right, should have included that information.

Let's say i have a local directory (from a git cloned on previous task in the plan) with:

```
./file1.ext

./file2.ext

/dir1/file1-dir1.ext

/dir-dont-upload/file1-dir-dont-upload.ext

./file3.ext
./.some_module_rc
```

My "Amazon S3 Object" task is configured to upload the content of this folder to a s3 bucket.

I've checked the "\[ x \]Use Ant patterns to select files" option, and in "**Source Local Path"**field i've tried a lot of combinations, right now i'm using.

```
**,<defaultexcludes echo="true" add="/dir-dont-upload/**"/>, 
```

i've also tried

```
**,<exclude add="/dir-dont-upload/**"/>, 
```

But it always uploads all the files.

If i remove the "\*\*" starting, it doesn't upload anything.

Also, is there a quicker way (like some sandboxes commandline with debug, i could use to test this, as always run the entire plan, is somewhat complicated.

Thanks for any suggestion.

-wm


author: Craig Castle-Mead
body: Hi Werner,

Are you able to share the ant pattern you're using, what result you're expecting and the result you're getting? This may be enough for the community to help diagnose and suggest a fix.

CCM


Tags: addon-net.utoolity.atlassian.bamboo.tasks-for-aws

aws

sync