How to configure Apache for Confluence 6.2 Synchrony cluster
Asked by Areg Vrtanesyan _Work_:

Hi  

I am wondering if someone has managed to configure Apache for Synchrony Cluster proxy in?  

I have well established configuration for Jira and Confluence DC cluster based on Apache 2.4 + MOD_JK + AjpNio connector.  

Now I would like to add Synchrony in it but I can't find out any sample on Atlassian for that.  

There is only one reference to single server option.  

Regards,  
Areg


Answers:
author: Areg Vrtanesyan _Work_
body: Hi All

With help of my collegaues, PS Yilin Mo and TAM Kevin we have solved this puzzle.

So the steps are involving:

* Use Apache 2.4.x
* Download, compile and install MOD_JK - <https://tomcat.apache.org/download-connectors.cgi>
* Configure and terminate SSL in Apache
* Configure MOD_JK for Confluence Nodes:

```

#===================== Worker Section Start =====================
# Confluence Load Balancer Worker
worker.list=confluence-lb
worker.confluence-lb.type=lb
worker.confluence-lb.balance_workers=confluence-n1,confluence-n2
worker.confluence-lb.sticky_session=true
worker.confluence-lb.session_cookie=JSESSIONID
# Worker for Confluence Node 1 - HTTPS
worker.confluence-n1.type=ajp13
worker.confluence-n1.ping_mode=I
worker.confluence-n1.socket_connect_timeout=5000
worker.confluence-n1.host=confluence-node1
worker.confluence-n1.port=8600
worker.confluence-n1.route=confluence-node1
# Worker for Confluence Node 2 - HTTPS
worker.confluence-n2.type=ajp13
worker.confluence-n2.ping_mode=I
worker.confluence-n2.socket_connect_timeout=5000
worker.confluence-n2.host=confluence-node2
worker.confluence-n2.port=8600
worker.confluence-n2.route=confluence-node2
#=====================? Worker Section End? =====================
```

* Configure Confluence to accept connections from MOD_JK

```
??????? <!-- Main Connector Port -->
??????? <Connector port="8600"

??????????? maxThreads="250"
??????????? minSpareThreads="25"
??????????? connectionTimeout="20000"

??????????? enableLookups="false"
??????????? protocol="org.apache.coyote.ajp.AjpNioProtocol"
??????????? redirectPort="8443"
??????????? acceptCount="10"
??????????? debug="0"
??????????? URIEncoding="UTF-8"

??????????? proxyName="confluence.something.com"
??????????? proxyPort="443"

??????? />
```

* Configure Apache for MOD_JK

```
??? JkMount?? ? /*?? ??? ???      ?confluence-lb
?? ?JkUnMount?? /html/*?? ??? ??? ?confluence-lb
?? ?JkUnMount?? /synchrony/*?? ??? confluence-lb
?? ?JkUnMount?? /balancer-manager? confluence-lb
```

* Configure Load Balancer and proxy for Synchrony. This has been worked out with Yilin help and long time digging in Internet. Finally with network debugging I have found that Synchrony url's ending with "/websocket" should be handled by proxy as Websocket connection not http. I have found nice simple explanation - <https://serverfault.com/questions/616370/configuring-apache-2-4-mod-proxy-wstunnel-for-socket-io-1-0>

```
??? <Location /balancer-manager>
?? ??? ?SetHandler balancer-manager

?? ??? ?Require?? ??? ??? ??? ?<access list>
?? ?</Location>

?? ?#Synchrony configuration
?? ?ProxyRequests Off
?? ?ProxyPreserveHost On

?? ?# Proxy Balancer
?? ?<Proxy balancer://confluence-synchrony-http>
?? ??? ?# Synchrony Node 1
?? ??? ?BalancerMember?? ??? ?http://synchrony-node1:9100 route=confluence-synnode1
?? ??? ?# Synchrony node 2
?? ??? ?BalancerMember?? ??? ?http://synchrony-node2:9100 route=confluence-synnode2

?? ??? ?# Security
?? ??? ?Require?? ??? ??? ??? ?all granted

?? ??? ?# Load Balancer Settings
?? ??? ?# We are not really balancing anything in this setup, but need to configure this
?? ??? ?ProxySet?? ??? ??? ?lbmethod=byrequests
?? ??? ?ProxySet?? ??? ??? ?stickysession=JSESSIONID
?? ?</Proxy>
?? ?<Proxy balancer://confluence-synchrony-ws>
?? ??? ?# Synchrony Node 1
?? ??? ?BalancerMember?? ??? ?ws://synchrony-node1:9100 route=confluence-synnode1
?? ??? ?# Synchrony Node 2
?? ??? ?BalancerMember?? ??? ?ws://synchrony-node2:9100 route=confluence-synnode1

?? ??? ?# Security
?? ??? ?Require?? ??? ??? ??? ?all granted

?? ??? ?# Load Balancer Settings
?? ??? ?# We are not really balancing anything in this setup, but need to configure this
?? ??? ?ProxySet?? ??? ??? ?lbmethod=byrequests
?? ??? ?ProxySet?? ??? ??? ?stickysession=JSESSIONID
?? ?</Proxy>

?? ?# https://serverfault.com/questions/616370/configuring-apache-2-4-mod-proxy-wstunnel-for-socket-io-1-0x
?? ?RewriteEngine?? ? On
?? ?RewriteCond?? ??? %{REQUEST_URI}?? ?? ?^/synchrony/(.*)/websocket?? ??? ??? ?[NC]
?? ?RewriteRule?? ??? /(.*)?? ??? ??? ??? ?balancer://confluence-synchrony-ws/$1? [P,L]
    RewriteCond       %{REQUEST_URI}?? ??? ^/synchrony/(.*)/bayeux-sync1?????????????????????????? [NC]
    RewriteRule?????? /(.*)??????????????? balancer://confluence-synchrony-pro0-ws/$1????? [P,L]    

?? ?#ProxyPass?? ??? ?/balancer-manager?? ?!
?? ?ProxyPass?? ??? ? /synchrony?? ??? ??? ?balancer://confluence-synchrony-http/synchrony
?? ?ProxyPassReverse? /synchrony?? ??? ??? ?balancer://confluence-synchrony-http/synchrony
```

* Start Confluence and Synchrony as in <https://confluence.atlassian.com/conf62/installing-confluence-data-center-902075269.html> with correct posrts set
* Remember to secure Synchrony servers as there is a security issues with the Synchrony command line - <https://jira.atlassian.com/browse/CONFSERVER-53085>
* Enable Collaborative Editing
* Enjoy

Hopefully this is all.

comments: author: Sean McManus
body: i created a DNS entry for the VIP on the load balancer in front of the two synchrony nodes. I changed my mod proxy vhost file to call the vip name.

e.g.

```
ProxyPass /synchrony https://synchrony.xxx.com/synchrony

    ProxyPassReverse /synchrony https://synchrony.xxx.com/synchrony 

    <Location /synchrony>

        Require all granted

        RewriteEngine on

        RewriteCond %{HTTP:UPGRADE} ^WebSocket$ [NC]

        RewriteCond %{HTTP:CONNECTION} Upgrade$ [NC]

        RewriteRule .* ws://synchrony.xxx.com%{REQUEST_URI} [P]

    </Location>
```

because i made the call over SSL i also had to add this to the vhost file.

```
    SSLProxyCheckPeerCN off

    SSLProxyCheckPeerName off
```


author: sarath sasi
body: Hi Sean,

How you accomplished ws load balancing.?

I have a F5 in front of two synchrony nodes it is load balancing http/s traffics but not ws .Do we have to enable some configuration in F5?


author: RJ Confluence Admins
body: I had to drop the SSL connection. =( I call the vip as 8091 and pass through now.


author: sarath sasi
body: Thanks.

Will that mean you put a rewwrite rule like this.

```
 RewriteRule .* ws://synchrony.xxx.com:8091%{REQUEST_URI} [P]


1.Is Ws traffic passing through F5 with this config?
2.Do we have to do any settings in F5 for this?
```


author: sarath sasi
body: That will be fine but my apache load balancing is configured only for synchrony servers as I have F5 for confluence load balancing.I had discussion with PS on this session stickiness part for synchrony.I guess for internal requests to synchrony JSESSION id will not be passed.So not sure apache sticky session with JSESSION ID makes sense for synchrony hub.

Will post my observation once I get a concrete evidence.

comments: author: sarath sasi
body: As per vendor update LB for synchrony doesn't need session stickiness.


author: sarath sasi
body: Hi All,

We are observing below behavior.

1.intermittently in one node suddenly saving will fail and it will fix itself after 5 min.

2.Browser will give 204 response code.

3.Logs are not giving much info.

Does anyone faced this issue?.We are also using apache for loadbalancing synchrony cluster.


author: sarath sasi
body: HI Areg,

I tried configuring the same in apache for synchrony hub.

Also i noticed that the synchrony traffic is wss:xxxx/synchrony

So i used the configuration for rewrite in apache

\<Location /synchrony\>

Require all granted

RewriteEngine on

RewriteCond %{HTTP:UPGRADE} \^WebSocket$ \[NC\]

RewriteCond %{HTTP:CONNECTION} Upgrade$ \[NC\]

RewriteRule /(.\*) balancer://confluence-synchrony-ws/$1 \[P,L\]

\</Location\>

But I am getting response code of 204 no content even though connection is getting success.

comments: author: Areg Vrtanesyan _Work_
body: Hi

Which version of Confluence you are running?

In latest versions it has changed.

You can check it by using Firefox network monitor.

I will post fixed configuration once will be online.


author: RJ Confluence Admins
body: What changed and in which version???  

<br />


author: Areg Vrtanesyan _Work_
body: In new version of Synchrony it is looking **++\^/synchrony/(.\*)/bayeux-sync1++** URL instead of **++\^/synchrony/(.\*)/websocket++** and it is not documented anywhere ...

It is possible to detect by usung Firefox or Chrome Network Monitor and see which request fails on Websocket and add rewrite condition for it.

I have updated my answer for Apache - working as a charm for us.


author: sarath sasi
body: Hi Areg,

Thanks for sharing this .I configured the apache to act as lb for synchrony hub.

Configured jsession id for session stickiness.Is there any way to confirm this is working or not?


author: RJ Confluence Admins
body: Use developer tools in chrome

application tab

expand the cookies and watch if the jsessionid cookie stays the same when running more than 1 app server.


author: RJ Confluence Admins
body: Slight hijack... and apologies for that but very related.  

I have synchrony behind it's pown VIP and just call it as 8091 and all that works well. My issue is for the app side. When I have more than 1 server on the VIP for the app side active, users could not check boxes or change status macros with out being in edit mode. I was able to fix this with changing to use the JSESSIONID cookie in the F5 in front of the app. It appears though that the Apache/mod_proxy server in front of the app tier is breaking the JSESSIONID cookie. For the life of me I can't figure out why / how. I am about to switch to NGiNX instead.


author: AnnWorley
body: In front of your Data Center cluster there should be a load balancer. Configure Apache in front of the IP being used for the cluster, using the same Apache settings you would use for the single server.

[This doc](https://confluence.atlassian.com/kb/proxying-atlassian-server-applications-with-apache-http-server-mod_proxy_http-806032611.html)has a sample config in an expand macro just under the text "If you're using Confluence 6.0 or later with Synchrony (which is required for collaborative editing), you'll need to use Apache 2.4.10 or later..."

comments: author: Areg Vrtanesyan _Work_
body: Hi Ann

That document doesn't describe how to do that for multiple synchrony instances running on a multiple nodes.

I know how to configure MOD_JK with Sticky Session for 2 node Confluence DC and it is working quite good.

Something like this <https://confluence.atlassian.com/confkb/how-to-configure-apache-mod_jk-to-proxy-confluence-6-x-or-later-867351565.html> but again it is for single host. How about multiple hosts?

Now when I have started to prepare for 6.2 update I have a feeling that there is a lack of documentation around it.

Regards,

Areg


author: Areg Vrtanesyan _Work_
body: Right

For the Datacenter Jira or Confluence you can do following:

\* with mod_proxy and mod_ajp

```
 # Proxy Balancer <Proxy balancer://confluence-cluster>  # Confluence Node 1  BalancerMember ajp://confluence-node1:8600 route=node1 timeout=1800  # Confluence Node 2  #BalancerMember ajp://confluence-node1:8600 route=node2 timeout=1800  # Security  Order  Deny,Allow  #Deny from None  Allow from All  # Load Balancer Settings  # We are not really balancing anything in this setup, but need to configure this  ProxySet lbmethod=byrequests  ProxySet stickysession=JSESSIONID </Proxy> ProxyRequests  Off ProxyPreserveHost On ProxyPass  /  balancer://confluence-cluster/ ProxyPassReverse /  balancer://confluence-cluster/
```

\* with mod_jk

"workers.properties" file

```
 #===================== Worker Section Start ===================== # Load Balancer Worker worker.list=confluence-lb worker.confluence-lb.type=lb worker.confluence-lb.balance_workers=confluence-node1,confluence-node2 worker.confluence-lb.sticky_session=true worker.confluence-lb.session_cookie=JSESSIONID # Worker for Node 1 - HTTPS worker.confluence-node1.type=ajp13 worker.confluence-node1.ping_mode=I worker.confluence-node1.socket_connect_timeout=5000 worker.confluence-node1.host=confluence-node1 worker.confluence-node1.port=8600 worker.confluence-node1.route=confluence-node1 # Worker for Node 2 - HTTPS worker.confluence-node2.type=ajp13 worker.confluence-node2.ping_mode=I worker.confluence-node2.socket_connect_timeout=5000 worker.confluence-node2.host=confluence-node2 worker.confluence-node2.port=8600 worker.confluence-node2.route=confluence-node2 #=====================  Worker Section End  =====================
```

Apache section

```
 JkMount    /*     confluence-lb
```

For Synchrony I can only see example in <https://confluence.atlassian.com/doc/installing-confluence-data-center-203603.html> I can only see section 6 for Synchrony like

```
6. Configure your load balancer for Synchrony.Your load balancer must support WebSockets (for example NGINX 1.3 or later, Apache httpd 2.4, IIS 8.0 or later) and session affinity. SSL connections must be terminated at your load balancer so that Synchrony can accept XHR requests from the web browser. 
```

and nothing

Then in <https://confluence.atlassian.com/confkb/how-to-configure-apache-mod_jk-to-proxy-confluence-6-x-or-later-867351565.html?_ga=2.209825301.1803497839.1495787573-1935709168.1489762288>

```
# Put this after the other LoadModule directivesLoadModule jk_module modules/mod_jk.so# Adding mod_proxy w/ wstunnel and mod_rewrite for Synchrony supportLoadModule proxy_module /usr/lib/apache2/modules/mod_proxy.soLoadModule proxy_http_module /usr/lib/apache2/modules/mod_proxy_http.soLoadModule proxy_wstunnel_module /usr/lib/apache2/modules/mod_proxy_wstunnel.soLoadModule rewrite_module /usr/lib/apache2/modules/mod_rewrite.so# Put this in the main section of your configuration (or virtual host, if using Apache virtual hosts)JkWorkersFile conf/workers.propertiesJkLogFile logs/apache2/mod_jk.logJkLogLevel info<VirtualHost *:80> #Synchrony configuration ProxyRequests Off ProxyPreserveHost On <Proxy *>     Require all granted </Proxy> ProxyPass /synchrony http://localhost:8091/synchrony <Location /synchrony>  Require all granted  RewriteEngine on  RewriteCond %{HTTP:UPGRADE} ^WebSocket$ [NC]  RewriteCond %{HTTP:CONNECTION} Upgrade$ [NC]  RewriteRule .* ws://localhost:8091%{REQUEST_URI} [P] </Location>  # mod_jk configuration JkMount /confluence worker1 JkMount /confluence/* worker1</VirtualHost>
```

I have also asked PS and TAM and looks like there is no exact example of 2 node and more Synchrony configuration on Apache or else even the section 5 in <https://confluence.atlassian.com/doc/installing-confluence-data-center-203603.html> tells to start process on each node.

Small section in <https://confluence.atlassian.com/confkb/how-to-configure-apache-mod_jk-to-proxy-confluence-6-x-or-later-867351565.html?_ga=2.209825301.1803497839.1495787573-1935709168.1489762288> tells how to do the WebSockets redirection:

```
 ProxyPass /synchrony http://localhost:8091/synchrony <Location /synchrony>  Require all granted  RewriteEngine on  RewriteCond %{HTTP:UPGRADE} ^WebSocket$ [NC]  RewriteCond %{HTTP:CONNECTION} Upgrade$ [NC]  RewriteRule .* ws://localhost:8091%{REQUEST_URI} [P] </Location>
```

But not how to load balance between the Synchrony nodes.

Regards,

Areg


author: AnnWorley
body: As you mentioned, you have a ticket open with Atlassian's Premier Support. It looks like they are still investigating. When you get a configuration that works, I hope you will come back to the Community and share an example.