Direct synchrony access behind reverse proxy not working
Asked by Robin Jansohn:

I'm trying to migrate our Confluence Server installation to run in a docker container and behind a reverse proxy (nginx).

I have a working setup but only if I do not try to configure nginx to directly contact synchrony.

nginx

```
server {
  listen 443 ssl;
  server_name confluencetest.mycompany.com;

  location / {
    proxy_set_header X-Forwarded-Host $host;
    proxy_set_header X-Forwarded-Server $host;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

    resolver 127.0.0.11 valid=30s;
    set $upstream_confluence confluence;
    proxy_pass http://$upstream_confluence:8090;
  }

  location /synchrony {
    proxy_set_header X-Forwarded-Host $host;
    proxy_set_header X-Forwarded-Server $host;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

    resolver 127.0.0.11 valid=30s;
    set $upstream_confluence confluence;
    proxy_pass http://$upstream_confluence:8091/synchrony;

    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection "Upgrade";
  }
}
```

nginx logs

```
172.28.26.29 - - [15/Jan/2021:14:36:11 +0000] "GET /synchrony/heartbeat?_=1610721370126 HTTP/1.1" 404 23 "https://confluencetest.mycompany.com/display/PRCD/TEST+Master" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.141 Safari/537.36 Edg/87.0.664.75" "-"
172.28.26.29 - - [15/Jan/2021:14:36:11 +0000] "GET /synchrony-proxy/heartbeat?_=1610721370127 HTTP/1.1" 404 23 "https://confluencetest.mycompany.com/display/PRCD/TEST+Master" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.141 Safari/537.36 Edg/87.0.664.75" "-"
```

atlassian-synchrony.log

```
2021-01-15 15:36:11,135 DEBUG [159:StdOutHandler [/opt/java/openjdk/bin/java]] 2021-01-15 14:36:11,135 INFO [async-dispatch-1] [synchrony.middleware.logging] {:request {:remote-addr "172.30.3.2", :uri "/synchrony", :request-method :get, :query-string nil, "x-forwarded-for" "172.28.26.29"}, :response {:status 404}}
2021-01-15 15:36:11,221 DEBUG [159:StdOutHandler [/opt/java/openjdk/bin/java]] 2021-01-15 14:36:11,220 INFO [async-dispatch-3] [synchrony.middleware.logging] {:request {:remote-addr "172.30.3.2", :uri "/synchrony", :request-method :get, :query-string nil, "x-forwarded-for" "172.28.26.29"}, :response {:status 404}}
```

404 response payload

```
{"message":"Not Found"}
```

As soon as I remove the "location /synchrony" block above it works (via synchrony-proxy).

Can anybody share some light what I'm doing wrong?


Answers:
author: Thiago Masutti
body: [@Robin Jansohn](/t5/user/viewprofilepage/user-id/575328)

I got it working as well, based on your configuration shared on the other thread.

The important part is how you configure the proxy_pass directive when using variables.

Without the additional **resolve** and **variables** configuration, this would be the nginx conf for Synchrony:

```
location /synchrony {
   client_max_body_size 100m;
   proxy_set_header X-Forwarded-Host $host;
   proxy_set_header X-Forwarded-Server $host;
   proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
   proxy_pass http://10.10.0.3:8091/synchrony;
   proxy_http_version 1.1;
   proxy_set_header Upgrade $http_upgrade;
   proxy_set_header Connection "Upgrade";
}
```

With the additional configuration you need in Nginx, then the Synchrony location conf should be similar to the below:

```
location /synchrony {
   client_max_body_size 100m;
   proxy_set_header X-Forwarded-Host $host;
   proxy_set_header X-Forwarded-Server $host;
   proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
   set $upstream_synchrony confluence-test:8091;
   proxy_pass http://$upstream_synchrony;
   proxy_http_version 1.1;
   proxy_set_header Upgrade $http_upgrade;
   proxy_set_header Connection "Upgrade";
}
```

The important portion is on the **proxy_pass** configuration where we don't need to add the **/synchrony** path.

Kind regards,  
Thiago Masutti

comments: author: Robin Jansohn
body: [@Thiago Masutti](/t5/user/viewprofilepage/user-id/1209618)

thanks again for helping me track down the problem even if it wasn't an Atlassian problem after all!

The reason why it didn't work when using the resolver is explained in the official docs (<https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_pass>):
> **When variables are used** in `proxy_pass`:
>
> ```
> location /name/ {    proxy_pass http://127.0.0.1$request_uri;}
> ```
>
> In this case, **if URI is specified in the directive, it is passed to the server as is**, replacing the original request URI.

While not super important it is still better to specify **location /synchrony/** because then requests to synchrony-proxy are not sent down the wrong route to synchrony. That is by the way the reason why it did not work at all in my original configuration and why I even got a 404 when sending the synchrony-proxy calls.


author: Dave Theodore [Coyote Creek Consulting]
body: What version of Confluence? Modern versions no longer need you to handle the Synchrony proxy. It's done natively in the Tomcat configuration.

comments: author: Robin Jansohn
body: We are using version 7.4.6. Tomcat native configuration cannot be the solution in this case IMO as all traffic is routed through the reverse proxy. Here's an excerpt from the confluence log where you can see how the synchrony URLs are configured. I don't see any problem there.

```
2021-01-19 08:24:09,190 INFO [ThreadPoolAsyncTaskExecutor::Thread 7] [plugins.synchrony.bootstrap.DefaultSynchronyProxyMonitor] <init> synchrony-proxy healthcheck url: http://127.0.0.1:8090/synchrony-proxy/healthcheck
2021-01-19 08:24:26,407 INFO [lifecycle:thread-11] [plugins.synchrony.bootstrap.DefaultSynchronyProcessManager] startup Starting Synchrony and enabling Collaborative Editing
2021-01-19 08:24:26,462 INFO [synchrony-interop-executor:thread-1] [plugins.synchrony.bootstrap.DefaultSynchronyProcessManager] isSynchronyProxyEnabled proxy port present: true
2021-01-19 08:24:26,937 INFO [synchrony-interop-executor:thread-1] [plugins.synchrony.bootstrap.DefaultSynchronyProcessManager] debugPrintEnvironment Synchrony working dir: /var/atlassian/application-data/confluence
2021-01-19 08:24:26,938 INFO [synchrony-interop-executor:thread-1] [plugins.synchrony.bootstrap.DefaultSynchronyProcessManager] debugPrintEnvironment /opt/java/openjdk/bin/java -classpath /opt/atlassian/confluence/temp/3.1.0-release-confluence_7.4-0239807a.jar:/opt/atlassian/confluence/confluence/WEB-INF/lib/ojdbc8-19.8.0.0.jar -Xss2048k -Xmx1g synchrony.core sql
2021-01-19 08:24:27,050 INFO [synchrony-interop-executor:thread-1] [plugins.synchrony.bootstrap.DefaultSynchronyProcessManager] isSynchronyProxyEnabled proxy port present: true
2021-01-19 08:24:27,054 INFO [synchrony-interop-executor:thread-1] [plugins.synchrony.bootstrap.DefaultSynchronyProcessManager] updateSynchronyConfiguration Synchrony External Base URL: https://confluencetest.mycompany.com/synchrony,https://confluencetest.mycompany.com/synchrony-proxy
2021-01-19 08:24:27,055 INFO [synchrony-interop-executor:thread-1] [plugins.synchrony.bootstrap.DefaultSynchronyProcessManager] updateSynchronyConfiguration Synchrony External Service URL: https://confluencetest.mycompany.com/synchrony/v1
2021-01-19 08:24:27,055 INFO [synchrony-interop-executor:thread-1] [plugins.synchrony.bootstrap.DefaultSynchronyProcessManager] updateSynchronyConfiguration Synchrony Internal Service URL: http://127.0.0.1:8091/synchrony/v1
2021-01-19 08:24:36,277 WARN [synchrony-interop-executor:thread-2] [plugins.synchrony.config.DefaultSynchronyConfigurationManager] enableSharedDrafts [Collab editing plugin] Enabling Shared Drafts
```


author: Robin Jansohn
body: We are using version 7.4.6. Tomcat native configuration cannot be the solution in this case as all traffic is routed through the reverse proxy. Here's an excerpt from the confluence log where you can see how the synchrony URLs are configured. I don't see any problem there.

```
2021-01-19 08:24:09,190 INFO [ThreadPoolAsyncTaskExecutor::Thread 7] [plugins.synchrony.bootstrap.DefaultSynchronyProxyMonitor] <init> synchrony-proxy healthcheck url: http://127.0.0.1:8090/synchrony-proxy/healthcheck
2021-01-19 08:24:26,407 INFO [lifecycle:thread-11] [plugins.synchrony.bootstrap.DefaultSynchronyProcessManager] startup Starting Synchrony and enabling Collaborative Editing
2021-01-19 08:24:26,462 INFO [synchrony-interop-executor:thread-1] [plugins.synchrony.bootstrap.DefaultSynchronyProcessManager] isSynchronyProxyEnabled proxy port present: true
2021-01-19 08:24:26,937 INFO [synchrony-interop-executor:thread-1] [plugins.synchrony.bootstrap.DefaultSynchronyProcessManager] debugPrintEnvironment Synchrony working dir: /var/atlassian/application-data/confluence
2021-01-19 08:24:26,938 INFO [synchrony-interop-executor:thread-1] [plugins.synchrony.bootstrap.DefaultSynchronyProcessManager] debugPrintEnvironment /opt/java/openjdk/bin/java -classpath /opt/atlassian/confluence/temp/3.1.0-release-confluence_7.4-0239807a.jar:/opt/atlassian/confluence/confluence/WEB-INF/lib/ojdbc8-19.8.0.0.jar -Xss2048k -Xmx1g synchrony.core sql
2021-01-19 08:24:27,050 INFO [synchrony-interop-executor:thread-1] [plugins.synchrony.bootstrap.DefaultSynchronyProcessManager] isSynchronyProxyEnabled proxy port present: true
2021-01-19 08:24:27,054 INFO [synchrony-interop-executor:thread-1] [plugins.synchrony.bootstrap.DefaultSynchronyProcessManager] updateSynchronyConfiguration Synchrony External Base URL: https://confluencetest.mycompany.com/synchrony,https://confluencetest.mycompany.com/synchrony-proxy
2021-01-19 08:24:27,055 INFO [synchrony-interop-executor:thread-1] [plugins.synchrony.bootstrap.DefaultSynchronyProcessManager] updateSynchronyConfiguration Synchrony External Service URL: https://confluencetest.mycompany.com/synchrony/v1
2021-01-19 08:24:27,055 INFO [synchrony-interop-executor:thread-1] [plugins.synchrony.bootstrap.DefaultSynchronyProcessManager] updateSynchronyConfiguration Synchrony Internal Service URL: http://127.0.0.1:8091/synchrony/v1
2021-01-19 08:24:36,277 WARN [synchrony-interop-executor:thread-2] [plugins.synchrony.config.DefaultSynchronyConfigurationManager] enableSharedDrafts [Collab editing plugin] Enabling Shared Drafts
```


author: Thiago Masutti
body: Hi [@Robin Jansohn](/t5/user/viewprofilepage/user-id/575328)

Even if you are passing all traffic through the NGINX reverse proxy, you could still rely on Tomcat's synchrony proxy functionality to internally route its traffic to Synchrony JVM.

I would agree that using a reverse proxy to perform this operation is more efficient, though.

Since you are running Confluence on a docker container, what is the Synchrony port you are exposing outside of the container?

Regards,  
Thiago Masutti


author: Robin Jansohn
body: > Even if you are passing all traffic through the NGINX reverse proxy, you could still rely on Tomcat's synchrony proxy functionality to internally route its traffic to Synchrony JVM.

That is exactly the scenario which works as I wrote in the opening post.
> I would agree that using a reverse proxy to perform this operation is more efficient, though.

This is what I'm trying to achieve and it fails.
> Since you are running Confluence on a docker container, what is the Synchrony port you are exposing outside of the container?

I haven't changed the default port, so 8091. But I do not expose it outside of the container. It is only reachable within the attached docker network so nginx and confluence container can reach each other. This way it is impossible to reach synchrony directly for an external client without passing through nginx.


author: Thiago Masutti
body: Hey [@Robin Jansohn](/t5/user/viewprofilepage/user-id/575328)

Thanks for the feedback.

Would you be willing to go on a screen share and work together on this?  
If so, [this URL](https://www.temporary-url.com/AB157) allows you to book some time in my agenda based on your best availability.  
When I receive the booking, I will update the invite with a Zoom call.

Note that this is a temporary URL and it will expire in 2 hours. :)

Regards,  
Thiago Masutti


author: Robin Jansohn
body: Hi [@Thiago Masutti](/t5/user/viewprofilepage/user-id/1209618)

I just packed up work yesterday after writing that post. We can do a screen share today if you have time.

Kind regards,  
Robin Jansohn


author: Thiago Masutti
body: Hi [@Robin Jansohn](/t5/user/viewprofilepage/user-id/575328)

That's fine. It seems we are on different timezones.

[This is a link](https://www.temporary-url.com/A5D8B) to Zoom call.

I will keep it running for the next hours, so feel free to join it so we can take a look at it together.

Regards,  
Thiago Masutti


author: Robin Jansohn
body: I don't think I'm allowed to use Zoom in my company. Any chance we can do a Microsoft Teams call?


author: Thiago Masutti
body: Sure.

Send me the MS Teams link and I will join


author: Robin Jansohn
body: EDIT: removed link


author: Robin Jansohn
body: [@Thiago Masutti](/t5/user/viewprofilepage/user-id/1209618)

I got it working now with the docker DNS resolver. Here is my nginx configuration:

```
client_max_body_size 100m;
proxy_set_header X-Forwarded-Host $host;
proxy_set_header X-Forwarded-Server $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_http_version 1.1;

server {
? listen 443 ssl;
? server_name confluencetest.mycompany.com;

? location / {
??? resolver 127.0.0.11 valid=30s;
??? set $upstream_confluence confluence;
    proxy_pass http://$upstream_confluence:8090;
? }

? location /synchrony/ {
    resolver 127.0.0.11 valid=30s;
    set $upstream_confluence confluence;
    proxy_pass http://$upstream_confluence:8091;

??? proxy_set_header Upgrade $http_upgrade;
??? proxy_set_header Connection "Upgrade";
? }
}
```


author: Thiago Masutti
body: [@Robin Jansohn](/t5/user/viewprofilepage/user-id/575328)

I'm almost sure that using **location /synchrony/** makes nginx to send all the requests to the default **/ location**, which will then use the Tomcat Synchrony Proxy.

You can confirm by looking at the developer tools and validate requests are sent to **/synchrony** and not to **/synchrony-proxy**.

I was able to reproduce the issue if I'm using the resolver directive in nginx, so I'm testing things here.

Regards,  
Thiago


Tags: confluence-server

nginx

server

synchrony