Confluence behind apache reverse proxy denies Collaborate
Asked by M_ Ludwig:

Hi Community,

we are running our Confluence behind an apache https proxy on a Debian9 Server. Confluence is the only app running on that machine.

However Collaborate stops working and can not reach its backend.

In the apache logs i can see following entries:

\[proxy:error\] \[pid 7851\] (111)Connection refused: AH00957: HTTP: attempt to connect to 127.0.0.1:8091 (localhost) failed

Port 8091 is up and running, but somehow the websocketconnect fails miserably.

Here is our Apache Config for the proxy:

ApacheConfig SSL for Confluence

#### confluence section  
\<VirtualHost \*:443\>  
TimeOut 1000  
ServerName confluence.our.site  
DirectoryIndex index.html

SSLEngine on  
#Certificates  
SSLCertificateFile /etc/apache2/ssl/our.site.crt  
SSLCertificateChainFile /etc/apache2/ssl/AlphaSSL_Intermediate_CA.pem  
SSLCertificateKeyFile /etc/apache2/ssl/our.site.key  
SSLCACertificateFile /etc/apache2/ssl/GlobalSign_Root_R1_CA.pem

SSLProxyEngine on  
RewriteEngine on  
ProxyRequests off  
ProxyPreserveHost on

RequestHeader set X-Forwarded-Proto "https"  
RequestHeader set X-Forwarded-Port "443"

LogLevel info ssl:debug  
ErrorLog ${APACHE_LOG_DIR}/our.site-ssl-error.log  
CustomLog ${APACHE_LOG_DIR}/our.site-ssl-access.log combined

\<Proxy \*\>  
Require all granted  
\</Proxy\>

ProxyPass /synchrony http://localhost:8091/synchrony  
\<Location /synchrony\>  
Require all granted  
RewriteEngine on  
RewriteCond %{HTTP:UPGRADE} \^WebSocket$ \[NC\]

RewriteCond %{HTTP:CONNECTION} Upgrade$ \[NC\]  
RewriteRule .\* ws://localhost:8091%{REQUEST_URI} \[P\]  
\</Location\>

ProxyPass / http://localhost:8090/  
ProxyPassReverse / http://localhost:8090/

\<Directory "/var/www/our.site"\>  
Options +FollowSymLinks +SymLinksIfOwnerMatch  
AllowOverride All  
Require all granted  
\</Directory\>

#Certbot  
ProxyPass /.well-known !  
ProxyPassReverse /.well-known !

#additional ssl  
SSLProtocol all -SSLv2 -SSLv3  
SSLHonorCipherOrder on  
SSLCompression off

\</VirtualHost\>  
###end of confluence section

#### balsamiq rtc section  
Listen our.site:9083  
\<VirtualHost \*:9083\>  
SSLEngine on  
#Certificates  
SSLCertificateFile /etc/apache2/ssl/our.site.crt  
SSLCertificateChainFile /etc/apache2/ssl/AlphaSSL_Intermediate_CA.pem  
SSLCertificateKeyFile /etc/apache2/ssl/our.site.key  
SSLCACertificateFile /etc/apache2/ssl/GlobalSign_Root_R1_CA.pem

ProxyRequests off  
ProxyPreserveHost On  
RewriteEngine on

\<Proxy \*\>  
Require all granted  
\</Proxy\>

ProxyPass / http://localhost:9083/  
RewriteEngine on  
RewriteCond %{HTTP:UPGRADE} \^WebSocket$ \[NC\]  
RewriteCond %{HTTP:CONNECTION} Upgrade$ \[NC\]  
RewriteRule .\* ws://localhost:9083%{REQUEST_URI} \[P\]  
\</VirtualHost\>  
###end of balsamiq rtc section

We also have a port 80 vhost which redirects to the https:

\<VirtualHost \*:80\>  
TimeOut 1000  
UseCanonicalName On  
ServerAdmin root@localhost  
ServerName confluence.our.site  
DocumentRoot "/var/www/our.site"  
DirectoryIndex index.html  
ProxyRequests Off

LogLevel info  
ErrorLog ${APACHE_LOG_DIR}/our.site-prod-error.log  
CustomLog ${APACHE_LOG_DIR}/our.site-access.log combined

\<Directory "/var/www/our.site"\>  
AllowOverride All  
Options +FollowSymLinks +SymLinksIfOwnerMatch  
Require all granted  
\</Directory\>

RewriteEngine on  
#RewriteCond %{THE_REQUEST} !/.well-known/acme-challenge/ \[NC\]  
RewriteCond %{SERVER_NAME} =confluence.solar-log.com  
RewriteRule \^ https://%{SERVER_NAME}%{REQUEST_URI} \[END,NE,R=permanent\]  
\</VirtualHost\>

our confluence server.xml:

\<Server port="8000" shutdown="SHUTDOWN" debug="0"\>  
\<Service name="Tomcat-Standalone"\>

\<Connector port="8090" connectionTimeout="20000" redirectPort="8443"  
address="127.0.0.1"  
maxThreads="48" minSpareThreads="10"  
enableLookups="false" acceptCount="10" debug="0" URIEncoding="UTF-8"  
protocol="org.apache.coyote.http11.Http11NioProtocol"  
scheme="https" secure="true" proxyName="confluence.our.site" proxyPort="443"/\>

\<Engine name="Standalone" defaultHost="localhost" debug="0"\>  
\<Host name="localhost" debug="0" appBase="webapps" unpackWARs="true" autoDeploy="false" startStopThreads="4"\>  
\<Context path="" docBase="../confluence" debug="0" reloadable="false" useHttpOnly="true"\>  
\<!-- Logging configuration for Confluence is specified in confluence/WEB-INF/classes/log4j.properties --\>  
\<Manager pathname=""/\>  
\<Valve className="org.apache.catalina.valves.StuckThreadDetectionValve" threshold="60"/\>  
\</Context\>

\<Context path="${confluence.context.path}/synchrony-proxy" docBase="../synchrony-proxy" debug="0"  
reloadable="false" useHttpOnly="true"\>  
\<Valve className="org.apache.catalina.valves.StuckThreadDetectionValve" threshold="60"/\>  
\</Context\>  
\</Host\>  
\</Engine\>  
\</Service\>  
\</Server\>

We are unable to open the ws sockets - this is also for balsamiq which also does not work.

In order to use confluence we had to stop collaborate completely. But this is not what we wish. We need collaborate working.

We have no running firewall on that server, but we have security tools rkhunter, aide, auditd, acct and lynis working.

Can please someone help?

Greetings,

Tux


Answers:
author: M_ Ludwig
body: Hi Alexis,

thank you for response.

Yes i assure that Port 8091 is reachable via telnet:

telnet localhost 8091  
Trying 127.0.0.1...  
Connected to localhost.  
Escape character is '\^\]'.  
\^\[9  
HTTP/1.1 414 Request-URI Too Long

text is emptyConnection closed by foreign host.

comments: author: M_ Ludwig
body: Hi there,

we still have those severe problems which caused us to stop collaborate and thus balsamiq also is not working since it also uses websockets over synchrony.

Telnetting to the machines synchrony port works just fine from the local machine and also from a distant machine.

Also a wget of the heartbeat which works over synchrony is working and getting a file back with heartbeat ok.

But if we start Collaborate and thus restarting syncrony, we see in the apache ssl error logs always the following entries:

\[proxy:error\] \[pid 17880\] (111)Connection refused: AH00957: WS: attempt to connect to 127.0.0.1:8091 (\*) failed

\[proxy_wstunnel:error\] \[pid 17880\] \[client 192.168.110.218:50896\] AH02452: failed to make connection to backend: localhost

\[proxy:error\] \[pid 19466\] (111)Connection refused: AH00957: HTTP: attempt to connect to 127.0.0.1:8091 (localhost) failed

\[proxy_http:error\] \[pid 19466\] \[client 192.168.110.218:50892\] AH01114: HTTP: failed to make connection to backend: localhost

For a test we installed confluence on another server (also Debian9 with same settings) with exactly the same configuration in confluence itself and apache - it works there like a charm. On our productive server we always see above nasty errors we can not find out where they are coming from.

Any help in that is appreciated.

Greetings,

Tux


author: Alexis Robert
body: Hi [@M_ Ludwig](/t5/user/viewprofilepage/user-id/2242935) ,

I have pretty much the same configuration (my Apache proxy runs on another VM but very similar otherwise). Here's my Apache config (I left out the SSL stuff) :

```
<Proxy *>
Require all granted
</Proxy>
ProxyPass /synchrony http://my.ip:8091/synchrony

<Location /synchrony>
Require all granted
RewriteEngine on
RewriteCond %{HTTP:UPGRADE} ^WebSocket$ [NC]
RewriteCond %{HTTP:CONNECTION} Upgrade$ [NC]
RewriteRule .* ws://my.ip:8091%{REQUEST_URI} [P]
</Location>

ProxyPass / http://my.ip:8090/
ProxyPassReverse / http://my.ip:8090/

<Location />
Require all granted
</Location>
```

Did you make sure that you're not blocking requests on 8091 port ? Are you able to telnet to that port for example ?

comments: author: Alexis Robert
body: Hi [@M_ Ludwig](/t5/user/viewprofilepage/user-id/2242935) ,

if you keep seeing the errors on your production servers, but the setup works fine on your test server, then the issue is definitely with your production server.

I would check again if there is not a SELinux or iptables policy preventing the connection to synchrony.


author: M_ Ludwig
body: Hi Alexis,

we have no iptables set up and no selinux running. SE is not even installed on the server.

The problem is also only located to Apache proxy/Synchrony both running on exactly this machine having problems to connect via localhost or 127.0.0.1.

Connecting to the synchrony port on localhost from a login shell of the server itself is working as a charm.

We checked everything twice and could not find a configuration different from our testsystem. This exactly is the reason why we are asking here - we simply have no idea where to search next.

Greetings,

Tux


Tags: confluence-server

server