migrating your data from Neo4j 3.5 to the new and fancy 4.1
Tags: neo4j
Published: 2020-10-30 19:23:48

The new Neo4j 4.1 release is full of awesome and desirable features that you probably want, multiple databases on one dbms being not the least of them. While the instructions for upgrading your entire install, dbms and data together, is well-documented, the process for migrating your `graph.db` data from 3.5.6 to 4.1 is a little trickier.

Here, we will cover the process of installing a fresh Neo4j 4.1 and then successfully moving your data from a 3.5.6 install to your new 4.1.

These instructions assume Linux and the examples are for Ubuntu 18.04lts running on an AWS ec2 (ami-0817d428a6fb68645).

## Prepare your server
The preparation steps for your ec2 (or other box) are quite minimal.

### Firewall
For the firewall you have, at a minimum, two ports that you have to address:

**Browser access:** port 7474 should be open to all the ips that need access to the Neo4j browser.
**Bolt access:** port 7687 serves bolt and should be open to all applications that will be calling your Neo4j.

###Java
Your old Neo4j 3.5.6 ran on Java 8. That will no longer work with version 4.1. You will need to have Java 11 installed.

```
sudo apt update
sudo apt install openjdk-11-jre-headless
```

Test with:

```
java -version
```

You should see some output similar to:

```
openjdk version "11.0.9" 2020-10-20
```

##Install Neo4j 
Neo4j can be installed through `apt`. First, we update our `sources.list` to include Neo4j. 
```
wget -O - https://debian.neo4j.com/neotechnology.gpg.key | sudo apt-key add -
echo 'deb https://debian.neo4j.com stable 4.1' | sudo tee -a /etc/apt/sources.list.d/neo4j.list
sudo apt-get update
```

Then it is a simple matter of using apt to install. Note that we explicitly set the version number here to 4.1.3
```
sudo apt-get install neo4j=1:4.1.3
```

Once Neo4j is installed, we set systemd to launch the daemon on boot:

```
sudo systemctl enable neo4j.service
```

##Configure Neo4j
We're not going to start the Neo4j daemon just yet. First we're going to do some very basic, but necessary configuration.

The configuration file is located at `/etc/neo4j/neo4j.conf`. Open it in your favourite text editor and make the following adjustments:

Uncomment these two lines:

```
dbms.allow_upgrade=true
dbms.connectors.default_listen_address=0.0.0.0
```

For reference, these two lines are located near lines 31 and 86 respectively.

The `allow_upgrade` setting tells Neo4j to upgrade the format of legacy databases if they are encountered. Since we will be moving over data from 3.5.6 later, this is definitely required.

We uncomment the `default_listen_address` line to allow Neo4j to serve remote connections. If you are only using Neo4j locally, it is not necessary (and not a good idea, generally) to uncomment this line.

Next, look around line 9 to find line that looks similar to:

```
dbms.default_database=neo4j
```

Neo4j 4+ can run multiple different databases on the same server. One of them has to be the 'default' database that is loaded when you first log in to the browser. This is where you set that. Change `neo4j` here to the name you want for the database you will be transferring over. For the rest of this document, our example database will be called `startrekepisodes`, ie.

```
dbms.default_database=startrekepisodes
```

##Optionally install apoc
'Awesome procedures on Cypher', or 'apoc' for short, is an add-on library full of useful commands. They are optional, but very useful and worthwhile installing. The install process is simply downloading the appropriate jar file and putting it into the 'plugins' directory.

```
cd ~
curl -L https://github.com/neo4j-contrib/neo4j-apoc-procedures/releases/download/4.1.0.2/apoc-4.1.0.2-all.jar -O
sudo mv apoc-4.1.0.2-all.jar /var/lib/neo4j/plugins
sudo chown neo4j:adm /var/lib/neo4j/plugins/apoc-4.1.0.2-all.jar
```

Note here that we downloaded a specific version of apoc (the latest at time of writing). It is worthwhile to look at the apoc releases page and choose the most recent stable version. 

https://github.com/neo4j-contrib/neo4j-apoc-procedures/releases


##Transfer your data from 3.5.6
On Neo4j 3.5 you have one database and it is stored in the `graph.db` directory and migrating data between installs was as simple as moving that directory.

This is not the case with Neo4j 4.1 due to naming constraints.

On your Neo4j 3.5.6 server, go to where your databases file is located and tar up the entirety of your databases folder:
```
cd /var/lib/neo4j/data
tar cf databases.tar databases
```

Then transfer that `databases.tar` archive to your new Neo4j 4.1 box.

On the new server with Neo4j 4.1, untar `databases.tar` in safe location, then replace the contents of your Neo4j's `databases` directory with the contents from the `databases` you got form 3.5.6.
```
cd /tmp
tar xf databases.tar
cd /var/lib/neo4j/data/databases
sudo rm -f ./*
sudo mv /tmp/databases/* ./
```

Once that's done, all that is required is to rename your `graph.db` directory to the name of the `default_database` you set in `neo4j.conf`. In this example, that is `startrekepisodes`
```
cd /var/lib/neo4j/data/databases
sudo mv graph.db startrekepisodes
```

Then we set the ownership of the files to `neo4j` for both user and group
```
sudo chown -R neo4j:neo4j ./*
```

And our data is ready for Neo4j to upgrade to the new format.

##Start Neo4j and confirm
Finally, we start the Neo4j daemon.
```
sudo systemctl start neo4j.service
```

It's a good idea to watch the debug log output while Neo4j is starting up:
```
tail -f /var/log/neo4j/debug.log
```

You should see at some point, the database upgrade happen. It will look similar to this:
```
2020-10-29 23:54:36.017+0000 INFO [o.n.k.i.s.DatabaseMigrator] [startrekepisodes] Starting upgrade of database
2020-10-29 23:54:37.724+0000 INFO [o.n.k.i.s.DatabaseMigrator] [startrekepisodes] Migrating Store files (1/6):
2020-10-29 23:54:41.532+0000 INFO [o.n.k.i.s.DatabaseMigrator] [startrekepisodes]   10% completed
2020-10-29 23:54:41.532+0000 INFO [o.n.k.i.s.DatabaseMigrator] [startrekepisodes]   20% completed
2020-10-29 23:54:41.532+0000 INFO [o.n.k.i.s.DatabaseMigrator] [startrekepisodes]   30% completed
```

##Take it for a spin
Once Neo4j is done staring, navigate in the browser to your new Neo4j, set your new password and confirm that your data is in your default `startrekepisodes` database.