Fix Found for heartbeat problem
Asked by Peter Kahn:

Hi, I'm addressing BSP-7948 issue so I can inform you of the cause and fix in an effort to assist others. Please feel free to close this immediately.

**Fix:**

JVM wrapper settings changes fix problem

**Approach**

* Setup 2 node system
* Setup bogus forever job that runs cmd shell using infinite loop
* Used stress tool to max server disk, cpu, memory - could not cause heart beat problem
* Reviewed Logs and found pattern (see description). Agent attempts to restart JVM and complains of DLL problem on windows
* Fix DLL problem by running all win agents with x86 jdk
* Fix JVM restart problem by increasing time for jvm ping {code}wrapper.java.command=d: Java x86 jdk1.6.0_26 bin java.exe

```
wrapper.ping.timeout=900wrapper.ping.interval=30
```

**References**

* https://answers.atlassian.com/questions/61674/remote-agent-crashing-when-being-farmed
* http://wrapper.tanukisoftware.com/doc/english/prop-ping-timeout.html

**Problem:**

Agent builds stop and results lost after missing heartbeat. for months this appears to be server problem. Re-evaluate

* Determine if we can replicate the problem with micro environment under load
* Check agent logs looking for a pattern

**Situation:**

2013-02-12 11:12:22,785 INFO ActiveMQ Session Task DefaultErrorHandler

Recording error: Agent General Purpose Windows X64 (vmwbuild-03) went offline while building TRUNK-QUICKCOMPILEWINDOWS-2876.

The build results will not be saved. : TRUNK-QUICKCOMPILEWINDOWS

**Agent**

ERROR \| wrapper \| 2013/02/12 11:08:54 \| JVM appears hung: Timed out waiting for signal from JVM.

ERROR \| wrapper \| 2013/02/12 11:08:55 \| JVM did not exit on request, terminated

STATUS \| wrapper \| 2013/02/12 11:09:10 \| Launching a JVM...

ERROR \| wrapper \| 2013/02/12 11:09:34 \| Startup failed: Timed out waiting for a signal from the JVM.

ERROR \| wrapper \| 2013/02/12 11:09:34 \| JVM did not exit on request, terminated

STATUS \| wrapper \| 2013/02/12 11:09:39 \| Launching a JVM...

ERROR \| wrapper \| 2013/02/12 11:10:08 \| Startup failed: Timed out waiting for a signal from the JVM.

ERROR \| wrapper \| 2013/02/12 11:10:08 \| JVM did not exit on request, terminated

STATUS \| wrapper \| 2013/02/12 11:10:13 \| Launching a JVM...

ERROR \| wrapper \| 2013/02/12 11:10:42 \| Startup failed: Timed out waiting for a signal from the JVM.

ERROR \| wrapper \| 2013/02/12 11:10:42 \| JVM did not exit on request, terminated

STATUS \| wrapper \| 2013/02/12 11:10:47 \| Launching a JVM...

ERROR \| wrapper \| 2013/02/12 11:11:16 \| Startup failed: Timed out waiting for a signal from the JVM.

ERROR \| wrapper \| 2013/02/12 11:11:16 \| JVM did not exit on request, terminated

STATUS \| wrapper \| 2013/02/12 11:11:21 \| Launching a JVM...

INFO \| jvm 105 \| 2013/02/12 11:11:31 \| Wrapper (Version 3.2.3) <http://wrapper.tanukisoftware.org>

INFO \| jvm 105 \| 2013/02/12 11:11:31 \| Copyright 1999-2006 Tanuki Software, Inc. All Rights Reserved.

...

INFO \| jvm 105 \| 2013/02/12 11:11:32 \| WARNING - Unable to load the Wrapper's native library 'wrapper.dll'.

INFO \| jvm 105 \| 2013/02/12 11:11:32 \| The file is located on the path at the following location but

INFO \| jvm 105 \| 2013/02/12 11:11:32 \| could not be loaded:

INFO \| jvm 105 \| 2013/02/12 11:11:32 \| d: bamboo-agent-home bin .. lib wrapper.dll

INFO \| jvm 105 \| 2013/02/12 11:11:32 \| Please verify that the file is readable by the current user

INFO \| jvm 105 \| 2013/02/12 11:11:32 \| and that the file has not been corrupted in any way.

INFO \| jvm 105 \| 2013/02/12 11:11:32 \| One common cause of this problem is running a 32-bit version

INFO \| jvm 105 \| 2013/02/12 11:11:32 \| of the Wrapper with a 64-bit version of Java, or vica versa.

INFO \| jvm 105 \| 2013/02/12 11:11:32 \| This is a 64-bit JVM.

INFO \| jvm 105 \| 2013/02/12 11:11:32 \| Reported cause:

INFO \| jvm 105 \| 2013/02/12 11:11:32 \| D: bamboo-agent-home lib wrapper.dll: Can't find dependent libraries

INFO \| jvm 105 \| 2013/02/12 11:11:32 \| System signals will not be handled correctly.

INFO \| jvm 105 \| 2013/02/12 11:11:32 \|

INFO \| jvm 105 \| 2013/02/12 11:11:41 \| Agent bootstrap using baseUrl: <http://vmbamboo-01:8085/bamb>

{code}


Answers:
author: Peter Kahn
body: Heartbeat problems can happen when the agent restarts the local JVM when it fails to respond. We can avoid these by relaxing the schedule for checking the JVM.

wrapper.ping.timeout=900

wrapper.ping.interval=30

**References**

* [https://answers.atlassian.com/questions/61674/remote-agent-crashing-when-being-farmed](../61674/remote-agent-crashing-when-being-farmed)
* <http://wrapper.tanukisoftware.com/doc/english/prop-ping-timeout.html>


Tags: remote-agent