Bamboo times out publishing artifact
Asked by Brandon Vulaj:

I have a Job that (tries to) publish an artifact that other Stages and Jobs are dependant upon. However, it seems to time out whenever it tries to publish the artifact.  

If it matters, the artifact matches `**/* `so that I can copy the sources / compiled sources along the pipeline without re-checking out or re-compiling. The artifact is only roughly 1mb.  

I've attached the log below. Although Bamboo will retry and repeat the process for \~30 minutes or so until it finally gives up.

```
2014-10-09 15:09:03,803 INFO [0-BAM::Elastic Agent on i-84b5f06a::Agent:pool-3-thread-1] [ExecuteBuildTask] Running post build plugin 'Artifact Copier'2014-10-09 15:09:03,805 INFO [0-BAM::Elastic Agent on i-84b5f06a::Agent:pool-3-thread-1] [BuildArtifactPostProcessor] Copying the build artifacts for build: HS-WS-MCL-1112014-10-09 15:09:03,823 INFO [0-BAM::Elastic Agent on i-84b5f06a::Agent:pool-3-thread-1] [AbstractArtifactManager] Publishing [HoS Sources + Compiled Sources] for HS-WS-MCL-111: 68 file(s) mat ching [**/*] in directory /mnt/bamboo-ebs/bamboo-agent/build-dir/HS-WS-MCL2014-10-09 15:10:03,865 INFO [0-BAM::Elastic Agent on i-84b5f06a::Agent:pool-3-thread-1] [DefaultHttpClient] I/O exception (java.net.SocketTimeoutException) caught when processing request: Rea d timed out2014-10-09 15:10:03,865 INFO [0-BAM::Elastic Agent on i-84b5f06a::Agent:pool-3-thread-1] [DefaultHttpClient] Retrying request2014-10-09 15:11:03,922 INFO [0-BAM::Elastic Agent on i-84b5f06a::Agent:pool-3-thread-1] [DefaultHttpClient] I/O exception (java.net.SocketTimeoutException) caught when processing request: Rea d timed out2014-10-09 15:11:03,922 INFO [0-BAM::Elastic Agent on i-84b5f06a::Agent:pool-3-thread-1] [DefaultHttpClient] Retrying request2014-10-09 15:11:37,472 ERROR [tunnellogger-thread] [TunnelAcceptor] [tunnelserver:26224-1-thread-343] Error while accepting tunnel connections.java.net.SocketException: Connection reset	at java.net.SocketInputStream.read(SocketInputStream.java:168)	at com.sun.net.ssl.internal.ssl.InputRecord.readFully(InputRecord.java:422)	at com.sun.net.ssl.internal.ssl.InputRecord.read(InputRecord.java:460)	at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:863)	at com.sun.net.ssl.internal.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1188)	at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:818)	at com.sun.net.ssl.internal.ssl.AppInputStream.read(AppInputStream.java:75)	at java.io.InputStream.read(InputStream.java:82)	at com.atlassian.tunnel.tunnel.server.TunnelAcceptor.run(TunnelAcceptor.java:63)	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)	at java.lang.Thread.run(Thread.java:662)
```

-----

It may be worth mentioning that the first job **does** succeed to publish the artifact if it's the *first* job run on a newly spun up instance. However, subsequent jobs can not process the artifact. Logs below.  

```
2014-10-09 17:55:12,544 INFO [0-BAM::Elastic Agent on i-6be6a785::Agent:pool-3-thread-1] [BuildAgentControllerImpl] Hours of Service - hos-webservice - Maven Unit Test Lifecycle #113 (HS-WS-MUTL-113) taken from queue.2014-10-09 17:55:12,545 INFO [0-BAM::Elastic Agent on i-6be6a785::Agent:pool-3-thread-1] [RemoteExecutionPhaseServiceImpl] Hours of Service - hos-webservice - Maven Unit Test Lifecycle #113 (HS-WS-MUTL-113) assigned to agent 50135252014-10-09 17:55:12,618 INFO [0-BAM::Elastic Agent on i-6be6a785::Agent:pool-3-thread-1] [DefaultBuildAgent] Changing context: null -&gt; HS-WS-MUTL-113 on Elastic Agent on i-6be6a785/57319f762014-10-09 17:55:12,786 INFO [0-BAM::Elastic Agent on i-6be6a785::Agent:pool-3-thread-1] [BuildAgentControllerImpl] Build Hours of Service - hos-webservice - Maven Unit Test Lifecycle #113 (HS-WS-MUTL-113) started building on agent Elastic Agent on i-6be6a7852014-10-09 17:55:12,786 INFO [0-BAM::Elastic Agent on i-6be6a785::Agent:pool-3-thread-1] [DefaultBuildAgent] Running build phase: com.atlassian.bamboo.v2.build.task.InitializeBuild2014-10-09 17:55:12,787 INFO [0-BAM::Elastic Agent on i-6be6a785::Agent:pool-3-thread-1] [RemoteExecutionPhaseServiceImpl] Hours of Service - hos-webservice - Maven Unit Test Lifecycle #113 (HS-WS-MUTL-113) VCS sync started2014-10-09 17:55:12,947 INFO [0-BAM::Elastic Agent on i-6be6a785::Agent:pool-3-thread-1] [CheckoutDirectoriesSnapshotHelper] Build working directory is /mnt/bamboo-ebs/bamboo-agent/build-dir/HS-WS-MUTL2014-10-09 17:55:12,948 INFO [0-BAM::Elastic Agent on i-6be6a785::Agent:pool-3-thread-1] [DefaultBuildAgent] Running build phase: com.atlassian.bamboo.build.pipeline.tasks.PrepareBuildTask2014-10-09 17:55:12,949 INFO [0-BAM::Elastic Agent on i-6be6a785::Agent:pool-3-thread-1] [PrepareBuildTask] Executing build Hours of Service - hos-webservice - Maven Unit Test Lifecycle #113 (HS-WS-MUTL-113)2014-10-09 17:55:12,949 INFO [0-BAM::Elastic Agent on i-6be6a785::Agent:pool-3-thread-1] [PrepareBuildTask] Preparing artifact for use: HoS Sources + Compiled Sources2014-10-09 17:55:39,205 ERROR [tunnellogger-thread] [TunnelAcceptor] [tunnelserver:26224-1-thread-8] Error while accepting tunnel connections.java.net.SocketException: Connection reset        at java.net.SocketInputStream.read(SocketInputStream.java:168)        at com.sun.net.ssl.internal.ssl.InputRecord.readFully(InputRecord.java:422)        at com.sun.net.ssl.internal.ssl.InputRecord.read(InputRecord.java:460)        at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:863)        at com.sun.net.ssl.internal.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1188)        at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:818)        at com.sun.net.ssl.internal.ssl.AppInputStream.read(AppInputStream.java:75)        at java.io.InputStream.read(InputStream.java:82)        at com.atlassian.tunnel.tunnel.server.TunnelAcceptor.run(TunnelAcceptor.java:63)        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)        at java.lang.Thread.run(Thread.java:662)2014-10-09 17:55:40,203 ERROR [tunnellogger-thread] [TunnelAcceptor] [tunnelserver:26224-1-thread-8] Error while accepting tunnel connections.java.net.SocketException: Connection reset        at java.net.SocketInputStream.read(SocketInputStream.java:168)        at com.sun.net.ssl.internal.ssl.InputRecord.readFully(InputRecord.java:422)        at com.sun.net.ssl.internal.ssl.InputRecord.read(InputRecord.java:460)        at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:863)        at com.sun.net.ssl.internal.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1188)        at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:818)        at com.sun.net.ssl.internal.ssl.AppInputStream.read(AppInputStream.java:75)        at java.io.InputStream.read(InputStream.java:82)        at com.atlassian.tunnel.tunnel.server.TunnelAcceptor.run(TunnelAcceptor.java:63)        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)        at java.lang.Thread.run(Thread.java:662)2014-10-09 17:56:12,995 INFO [0-BAM::Elastic Agent on i-6be6a785::Agent:pool-3-thread-1] [DefaultHttpClient] I/O exception (java.net.SocketTimeoutException) caught when processing request: Read timed out2014-10-09 17:56:12,995 INFO [0-BAM::Elastic Agent on i-6be6a785::Agent:pool-3-thread-1] [DefaultHttpClient] Retrying request
```


Answers:
author: Pawel Skierczynski
body: If you have EIP on elastic agent and it will change IP after agent started it may be a cause of a problem.

If that's the case setting it up before agent starts could help.

This one is a little outdated but take a look at comments section.

<https://confluence.atlassian.com/display/BAMKB/Automatically+associating+Elastic+IP+addresses+to+Elastic+Agents>


author: Pawel Skierczynski
body: It's not very efficient to do it if there are many small files. How many files are there? If there are many could you try to tar/zip it before?

Have in mind that every job can be run on separate agent and in parallel in one stage. Also jobs in another stages don't have to run on same agents. So it's copied through the network both ways. I think it will also index information of every single file on a server side to database (but I may be wrong here).

Usually you would get sources from repository and dependant binaries through some kind of repository (like artifactory). It's all case by case basis so it's hard to say without knowing your exact pipeline.

comments: author: Brandon Vulaj
body: Hey Pawel,So I only have one agent running at the moment. There are 142 files for about 1.5mb.The pipeline looks like this. Maybe that gives you some insight as to why I'm sharing the artifact in this way.<http://i.imgur.com/2XfnULY.png>


author: Brandon Vulaj
body: If you have another suggestion, please let me know, but I would also like to solve this issue.


author: Brandon Vulaj
body: It's worth noting that I think this captures the spirit of what the Best Practices recommend: <https://confluence.atlassian.com/display/BAMBOO/Bamboo+Best+Practice+-+Sharing+artifacts>


author: Pawel Skierczynski
body: You may have one agent but Bamboo does not have one agent detection and any optimization for it. It works general case.What if all those were in one job as consecutive tasks? You would have one stage and one job. Admittedly you are loosing parallel running unit tests and integration tests. But unit tests are usually very quick and having one agent you don't use it at the moment anyway.You probably have more use cases to cover (everyone does) but they are not revealed by this diagram.


author: Pawel Skierczynski
body: Those best practices would better work with only few artifacts shared. Like if you build jar in stage one and them do some integration tests with it. (I personally would try the simplest possible thing that works so one stage, one job until you know you need more)Still 142 files doesn't seem that bad. You could try zip/make artifact/unzip only to see if it helps so it narrows down possible cause.There is still possibility that something else is wrong. Do you have OnDemand instance of server? What OS is Server and Agent running on? Any firewalls/proxies/other strange things worth mentioning like virtual machines/docker etc.?


author: Brandon Vulaj
body: I'll try the simple solution for now, which shouldn't require any artifact sharing at that level. I'm fairly positive it will work just fine.To fill you in, we are running OnDemand, we are behind a VPC and we do have EIP enabled. It is one of the default AMIs: ami-25f7ed4c. I see the comment at the bottom there - that seems like a potential solution. I'll try that and get back to you.


author: Pawel Skierczynski
body: I'll try to get to author of best practices. I believe that unit tests are usually done with source code so this diagram could be made better.


Tags: cloud