Confluence data center upgrade, cannot run with two different version at the same time
Asked by brbojorque:

We want to do a rolling upgrade (basically running two Confluence with different version at the same time) on Confluence Data Center but we cannot start the service because of this error.

<https://confluence.atlassian.com/confkb/when-starting-confluence-data-center-you-get-an-exception-bootstrapping-cluster-due-to-different-versions-911180880.html>

Can someone enlighten me why its not possible to do a rolling upgrade?


Answers:
author: crf
body: I only work on Jira, and these days only on Jira Cloud, so I can't provide an authoritative answer here. However, I will hazard the guess that Confluence Data Center has not implemented doing this because it is both complicated and risky to do so.

Normally, when we change the schema, this brings with it changes to all of the surrounding code that now understands the new model. If you are guaranteed that the schema is updated before the application starts accepting requests, then all of the code that works with the database, and all higher level code that deals with relationships with other database objects, only has to deal with a single situation.

To deal with such upgrades on a running system, we have to introduce a \*lot\* of complexity, it has to pass through several stages:

1. Where we started from, with version N of the schema, data, and code.
2. Version N schema and data are still being used, but the code has been upgraded to version N+1 and is using switching logic to do things the old version N way. This applies to everything from the schema, to your persistence logic, to any cluster-wide caching mechanisms that you might have, to any services that consider any newly existing relationships that didn't exist before, including both at the P2 level and in REST APIs.
3. Version N+1 schema updates are rolled out, but without any constraints that would prevent this, such as NOT NULL constraints on columns that we don't have data for yet. The code continues to operate in version N compatibility mode.
4. Version N+1 data writing is enabled, and existing data is upgraded, but code must still tolerate being able to read version N data while the upgrade is still in progress.
5. Any withheld constraints can finally be applied and the code switches to running in N+1 mode only.

Doing this just once is hard enough, but imagine the amount of work that's involved to do this for every single schema change. Further, any external caches that hold data objects have to somehow be invalidated before we reach step 5, as they can still be holding onto version N data that is no longer acceptable. Similar constraints also have to be placed on any P2 plugins you might have, which we don't actually control.

Now consider doing that for all of the schema changes between the current version and however far back we support doing an upgrade from. And then we have to clean all of this complexity back up later without breaking anything. And how do we make sure that you've actually completed this process before we start this version N to version N+1 process? What if you were running version N, but some of your data was still at N-1?

This is actually pretty close to what we do for Cloud, but the difference is that we completely control those deployments and live upgrade processes, so we can keep all of the scaffolding for these rollouts very focused and short-lived. The Server/Data Center release cycle problem and the fact that we don't control the exact timing and deployment of upgrades makes this much harder to carry off.

In short, while it's possible to do something like this, it requires so much work, is so fragile, and would still require so many other restrictions (such as saying that you only get live upgrades from one minor release to the next minor release) that the investment is probably very difficult to justify.

Then again... As I said I work on Jira Cloud, not Confluence Data Center, so this is just my guess.

(Note: brilliant as he is, Nic works for Adaptavist, not Atlassian ;) )

comments: author: brbojorque
body: So, there is no way to do an HA rolling upgrade for Confluence? The point of doing this is to provide an always online system while upgrading.

At least I can report back that it's not doable. In JIRA however it works 100% because of the feature "Upgrade Mode". I thought the "Read Only Mode" feature already solves this in Confluence.

Thanks for the very detailed information.


author: crf
body: Read-only mode helps solve many of these problems, but it is not sufficient on its own. The problem of being able to deal with two different data shapes with different sets of constraints on the data and different feature sets is still non-trivial, and it is also difficult to change this contract with the ecosystem vendors, as there would have to be changes that P2 plugin vendors would have to make adjustments for.

As I work on Jira Cloud, I'm not terribly familiar with how the Jira Data Center team dealt with this issue or how well it translates to the world of Confluence. Maybe it shows them a good way forward, or maybe some of the techniques involved just wouldn't work because of some significant difference in how Confluence works. I just don't know.

I would suggest that you watch/vote/comment on <https://jira.atlassian.com/browse/CONFSERVER-52686> which seems to be where that team is gathering interest around this requested feature. Although it is not the only motivating factor, the level of interest shown by the community is one of the factors taken into account when prioritizing feature development, so expressing your interest there is the best way to make sure that your own voice is counted.


author: brbojorque
body: Thanks [@crf](/t5/user/viewprofilepage/user-id/781494), Added your detailed information which may help them. Once again awesome explanation!


author: brbojorque
body: [@AnnWorley](/t5/user/viewprofilepage/user-id/502040), [@Yohaan Sunny](/t5/user/viewprofilepage/user-id/1629615), [@crf](/t5/user/viewprofilepage/user-id/781494), [@Neta Elyakim](/t5/user/viewprofilepage/user-id/1146705), [@Nic Brough -Adaptavist-](/t5/user/viewprofilepage/user-id/781454)

I'm summoning the best people in Atlassian to provide additional insights. :D


Tags: confluence-server

server