Errors after upgrading Confluence Data Center 7.19 to 8.5
Asked by Tim Sch?der:

Hi,

after upgrading Confluence DataCenter from 7.19 to 8.5 running with Windows Server 2019 64bit we experience the following problems :

- the Confluence-Windowsservice will not stop normally

- we get http500 errors on one node of the Confluence-Cluster

The following lines appear ..

ERROR \[https-jsse-nio2-8443-exec-22\] \[ContainerBase.\[Standalone\].\[localhost\].\[/\]\] log Unhandled exception occurred whilst decorating page  
-- url: / \| userName: xxxxxxx \| traceId: f37bc8698694e3f9  
java.lang.RuntimeException: javax.servlet.ServletException: org.apache.velocity.exception.VelocityException: ASTIdentifier() : exception invoking method for identifier 'rawOffset' in class sun.util.calendar.ZoneInfo  
Caused by: javax.servlet.ServletException: org.apache.velocity.exception.VelocityException: ASTIdentifier() : exception invoking method for identifier 'rawOffset' in class sun.util.calendar.ZoneInfo  
Caused by: org.apache.velocity.exception.VelocityException: ASTIdentifier() : exception invoking method for identifier 'rawOffset' in class sun.util.calendar.ZoneInfo  
Caused by: java.lang.IllegalAccessException: class org.apache.velocity.runtime.parser.node.PropertyExecutor cannot access class sun.util.calendar.ZoneInfo (in module java.base) because module java.base does not export sun.util.calendar to unnamed module @65ea0089  
2023-09-01 09:36:15,123 ERROR \[https-jsse-nio2-8443-exec-22\] \[\[Standalone\].\[localhost\].\[/\].\[default\]\] log Servlet.service() for servlet \[default\] in context with path \[\] threw exception  
java.lang.RuntimeException: javax.servlet.ServletException: org.apache.velocity.exception.VelocityException: ASTIdentifier() : exception invoking method for identifier 'rawOffset' in class sun.util.calendar.ZoneInfo  
Caused by: javax.servlet.ServletException: org.apache.velocity.exception.VelocityException: ASTIdentifier() : exception invoking method for identifier 'rawOffset' in class sun.util.calendar.ZoneInfo  
Caused by: org.apache.velocity.exception.VelocityException: ASTIdentifier() : exception invoking method for identifier 'rawOffset' in class sun.util.calendar.ZoneInfo  
2023-09-01 09:28:32,515 ERROR \[Catalina-utility-2\] \[atlassian.plugin.parsers.XmlDescriptorParser\] createModuleDescriptor There were problems loading the module 'job'. The module and its plugin have been disabled.  
2023-09-01 09:28:32,515 ERROR \[Catalina-utility-2\] \[atlassian.plugin.parsers.XmlDescriptorParser\] createModuleDescriptor There was a problem loading the descriptor for module 'job' in plugin 'Confluence JIRA Metadata Plugin'.\\n Job module descriptors are no longer supported. Use JobConfig modules instead.

Any suggestions?  
Kind regards

Tim

Update:

With help by Atlassian I solved the problem :

Add the following to your setenv.bat :

`set CATALINA_OPTS`="--add-exports java.naming/com.sun.jndi.ldap=ALL-UNNAMED ${CATALINA_OPTS}"  

Add the following to your JAVA-Properties:

*--add-opens=java.base/sun.util.calendar=ALL-UNNAMED*


Answers:
author: Bernhard Pressler
body: The problem origins in the reuse of the 7.19 setenv.sh or setenv.bat.

I used the one delivered with 8.5.x and it works now.

comments: author: Tim Sch?der
body: Hi Berhard, I try that ... as the copied installation will probably use the parameters set up with setenv.bat the 7.19-way ...

Thanks


author: Bernhard Pressler
body: As Confluence 8.5.x uses JDK17 there may be specific differences


author: Benjamin W_
body: Hi all

we also stumbled upon this issue (Windows Server 2019, Upgrade Confluence 7.19.5 to 8.5.0).

For us, using 8.5.1 did not solve the issue at all. Some errors disappeared, but Confluence did not start eitherway.

We also tried making use of JRE / JDK, but this showed not difference.

First when again using JDK 11.0.14.1_1 (instead of 17.0.7_7) we were able to successfully launch Confluence 8.5.1.

Regards

comments: author: Bernhard Pressler
body: I Think it is the same issue i had - on the upgrade the setenv.bat is not replaced by the new one. So do a clean 8.5.x installation on a test-environment without the ongoing setup-process and pick the new, fresh setenv.bat file for you upgraded environment.


author: Tim Sch?der
body: With help by Atlassian I solved the problem :

Add the following to your setenv.bat :

`set CATALINA_OPTS`="--add-exports java.naming/com.sun.jndi.ldap=ALL-UNNAMED ${CATALINA_OPTS}"  

Add the following to your JAVA-Properties:

*--add-opens=java.base/sun.util.calendar=ALL-UNNAMED*


author: Bernhard Pressler
body: The new setenv.bat / setenv.sh includes another file:

# Add various JPMS arguments to allow Confluence to work on Java 17  
CATALINA_OPTS="@$CATALINA_HOME/confluence/WEB-INF/jpms-args.txt ${CATALINA_OPTS}"

inside there are those parameters even shown by Tim

# Add various JPMS arguments to allow Confluence to work on Java 17.  
# These also work with Java 11, but are optional.  
--add-opens java.base/java.lang=ALL-UNNAMED  
--add-opens java.base/java.lang.reflect=ALL-UNNAMED  
--add-opens java.base/java.net=ALL-UNNAMED  
--add-opens java.base/java.io=ALL-UNNAMED  
--add-opens java.base/java.util=ALL-UNNAMED  
--add-opens java.base/java.util.concurrent.atomic=ALL-UNNAMED  
--add-opens java.base/sun.net.[www.protocol.jar=ALL-UNNAMED](http://www.protocol.jar=ALL-UNNAMED)  
--add-opens java.base/sun.net.[www.protocol.http=ALL-UNNAMED](http://www.protocol.http=ALL-UNNAMED)  
--add-opens java.base/sun.net.[www.protocol.https=ALL-UNNAMED](http://www.protocol.https=ALL-UNNAMED)  
--add-opens java.base/sun.util.locale=ALL-UNNAMED  
--add-opens java.management/javax.management=ALL-UNNAMED  
--add-exports java.base/sun.security.action=ALL-UNNAMED  
--add-exports java.base/sun.util.calendar=ALL-UNNAMED  
--add-exports java.naming/com.sun.jndi.ldap=ALL-UNNAMED  
--add-exports java.xml/com.sun.org.apache.xml.internal.utils=ALL-UNNAMED  
--add-exports java.desktop/sun.font=ALL-UNNAMED  
--add-exports java.base/sun.security.util=ALL-UNNAMED


author: Benjamin W_
body: Thank you very much for the fast feedback.

I can confirm that with those hints I was able to run Confluence 8.5.1 on Java 17.

As we run Confluence as a service (Windows), we added the parameters from %CATALINA_HOME%\\confluence\\WEB-INF\\jpms-args.txt onto the registry under Computer\\HKEY_LOCAL_MACHINE\\SOFTWARE\\WOW6432Node\\Apache Software Foundation\\Procrun 2.0\\Confluence\\Parameters\\Java\\Options9


author: Aron Gombas _Midori_
body: By the way, the problem is caused by the \<job\> definition in the app called "Confluence JIRA Metadata Plugin".

You may want to try if you can start Confluence without this app. (There is some commandline parameter to start up Confluence without certain apps.)


author: Aron Gombas _Midori_
body: The culprit here is the \`sun.util.calendar.ZoneInfo\` class. It is a Java class part of the Java platform and the corresponding time management packages went through changes from one Java version to another.

The first thing you should check is whether your particular Confluence version is running on a recommended Java version and distribution!

comments: author: Tim Sch?der
body: As I installed via the Windows installer I would think that this should work... both nodes are running with  

openjdk version "17.0.7" 2023-04-18  
OpenJDK Runtime Environment Temurin-17.0.7+7 (build 17.0.7+7)  
OpenJDK 64-Bit Server VM Temurin-17.0.7+7 (build 17.0.7+7, mixed mode, sharing)


author: Aron Gombas _Midori_
body: And this is what you see in the Jira admin screen (in the web UI)?

I think it is quite typical on Windows that there are multiple JREs installed, and a Java program is executed with an unexpected JRE. Therefore, the best is if you check it in the Jira admin screen.


author: Tim Sch?der
body: Checking the JVM-Parameters in the Jira admin screen does not sho any differences or problems. THe used version is still 17.0.07 running in sun.boot.library.path Confluence-Home\\jre\\bin.

There is just one java.exe installed on the whole system...


author: Aron Gombas _Midori_
body: OK, then this one is eliminate! Check the hint in my other answer.


author: Tim Sch?der
body: Sorry, I don't get it. I should delete the one and only java.exe an the system?? If this is really the causing problem, why is the first node in this dataCenter-Cluster running without any problems and the copied one expereinces such problems?


author: Aron Gombas _Midori_
body: No, sorry. I said that Java version is "eliminated" (removed, ruled out) from the potential root causes.

And I suggest that you see my other answer above. As you see, from the stack trace, it can be identified in which app the problem comes up. And I remember that Jira can be started with disabling certain apps. I think it would worth a try to start Jira with the problematic app turned off.

It is not a solution, but an experiment to validate if that app is the problem and there is nothing else.

Does that make sense?


author: Tim Sch?der
body: Ah ... sorry for misunderstanding :-) ... and we are talking about Confluence ... I check to disable this App.

:-)


Tags: atl-bobross

confluence-server

server