How do I script the exporting and importing of spaces?
Asked by Brian M Thomas:

I need to export over 500 Confluence spaces, and I don't have weeks to do it. Na?vely, I thought that I could use the remote interfaces to do it, but the REST API doesn't have the function and the XMLRPC version has been turned off. I am assured that the SOAP version of the same method has somehow survived, but the time required to create a simple SOAP client is more than I can spend, because I don't know how to do it, and my efforts to learn how have not yet borne fruit.

So I'm back to trying to script the UI. I get some encouraging results using a browser extension that lets me compose the requests and send them, but so far, I haven't managed to get them to work.

I sent a POST request to **/spaces/doExportSpace.action** with the values sent by the form presented by **/spaces/exportspacexml.action**, but I get a permission-denied response.

The browser is already logged in, so I'm guessing that there's something I'm failing to copy from the login response to my export request. So what is that? I've taken note of a cookie called "seraph.confluence" and the hidden fields in the request form.

I'm getting pretty desperate here; I've taken a week trying to figure out something that should have been an hour's work.

What can you tell me? The objective here is to be able to script hundreds of exports; I've tried everything anyone has suggested in hours of searching, but without success.


Answers:
author: Frank Liang
body: Is there a CLOUD Confluence export all spaces script?


author: Stephen Deutsch
body: Hi Brian,

I know it's a really late answer, but I hadn't forgotten, it just took longer than I expected to be able to test properly. I am also including it in case someone else needs to do the same thing.

I wrote a script in Python (version \>3.0) that allows one to be able to export all spaces:

```
import requests
import shutil
from time import sleep
from requests import Session
from requests.auth import HTTPBasicAuth
from zeep import Client
from zeep.transports import Transport

user = "admin"
password = "admin"

session = Session()
session.auth = HTTPBasicAuth(user, password)
client = Client('http://localhost:1990/confluence/rpc/soap-axis/confluenceservice-v2?WSDL',
    transport=Transport(session=session))

print("Logging in...")
token = client.service.login(user, password)
print("Getting Spaces")
spaces = client.service.getSpaces(token)
numSpaces = len(spaces)
for index, space in enumerate(spaces):
    print("Exporting space {} of {} - {} using URL:".format(index+1, numSpaces, space.key))
    siteExportUrl = client.service.exportSpace(token, space.key, "TYPE_XML")
    print(siteExportUrl)
    filename = siteExportUrl.split('/')[-1]
    print("Saving as: " + filename)
    r = session.get(siteExportUrl, stream=True)
    with open(filename, 'wb') as f:
        shutil.copyfileobj(r.raw, f)
    sleep(5)
```

You will need to install zeep before executing, but that should be the only dependency. Just change out the username and password, and change localhost:1990 to the base url of your confluence instance. The script pauses between each space for 5 seconds so as not to overload your server too much.

comments: author: Brian M Thomas
body: Thanks; I'll give it a try.


author: Krishna Ponnekanti
body: Able to export spaces. Need to verify if content and comments , etc are part of space or not


author: Brian M Thomas
body: I assume that the following assumptions are true of your comment:

* In the first sentence (which is incomplete, lacking both subject and verb), you are saying that *at present,* *you* are able to export spaces. Alternatively, that *someone* can do it now.
* Second, you want to know whether the "content and comments , etc" are part of the data in the *export*. Seems obvious, because otherwise, it wouldn't be clear which items you're talking about, but it isn't literally what you asked.

Those are the most likely probabilities, but the question could admit of other interpretations.

To the first sentence, which by my assumptions is not a question but a statement of fact, I can affirm that it is possible for some person(s) to export the file, though you don't specify that it can be scripted, which is the issue to which you've posted your reply, so that's not absolutely certain.

To the second, I will confirm that all data pertinent to the space and its content -- including, among other things, the content of every page, blog post, comment, attachment, label, and every extant version of the above, along with the user IDs of those who added them and the time they were added, is indeed in the export file. But this only happens when you export the *entire* space, using the *XML export* option, which is mostly explained in the admin interface where the export is requested.

If these assumptions are wrong, please clarify; also, if you want an answer to the question as I have interpreted it, it would be best if you raised another support issue since, not being relevant to this one, it may be overlooked.


author: Michael Stella
body: FYI, this SOAP client doesn't work, see Brian's answer in the conversation above.

Amazing that there's just no way to do this short of mechanical web clients like Selenium or Mechanize. Ridiculous.


author: Brian M Thomas
body: Indeed; in case I didn't make it clear as I thought I did, it doesn't matter which, or what kind, of SOAP client you use, it won't work because the method no longer exists.

I should add that the delay while I tried to figure out a way to do this caused my client to abandon the project altogether. I got paid for the work already done, but nothing more.


author: Michael Stella
body: Quite. We're realizing that for this reason and many more, perhaps Confluence is not the tool for us. There have been a lot of reliability issues recently with the cloud-hosted service, which got me trying to create a local mirror for when the cloud-hosted site is offline. Oh well.


author: Nic Brough -Adaptavist-
body: Rather than a local mirror, have you considered just moving to Server?


author: Michael Stella
body: For the price they want? Haha, no.


author: Krishna Ponnekanti
body: I followed the script given by Stephen Deutsch. I am able to export all the spaces in Confluence. But i cloud see a diff when compared with manually downloaded space ( space tools --\> contents and tools --\> Export --\> select XML and full export). in few spaces attachments missed and in few some diff in entities.xml file)

Is there any working way to export all spaces without loosing any content?


author: Stan Ry
body: Hello,

Thanks for sharing the script.

I am trying to invoke it, but get errors from Zeep module when invoking exoportSpace() method.

Here's the error:

```
~\AppData\Local\Continuum\anaconda3\lib\site-packages\zeep\xsd\elements\element.py in validate(self, value, render_path)
246 if not self.is_optional and not self.nillable and value in (None, NotSet):
247 raise exceptions.ValidationError(
--> 248 "Missing element %s" % (self.name), path=render_path)
ValidationError: Missing element in3 (exportSpace.in3)
```

Slightly modified script I am using:

```
import getpass
import requests
import shutil
from time import sleep
from requests import Session
from requests.auth import HTTPBasicAuth
from zeep import Client
from zeep.transports import Transport

requests.packages.urllib3.disable_warnings()
auth = ('my_account', getpass.getpass())
session = Session()
session.verify = False;
session.auth = auth;


WIKI = 'https://confluence.example.net/instance/rpc/soap-axis/confluenceservice-v2?WSDL'
client = Client(WIKI, transport=Transport(session=session))
print("Logging in...")
token = client.service.login(auth[0], auth[1])

#-Testing exportSpace() on personal space
print("Exporting personal space...")
numSpaces = 1
for index, space in enumerate(spaces):
print("Exporting space {} of {} - {} using URL:".format(index+1, numSpaces, spacekey))
print(spacekey);
siteExportUrl = client.service.exportSpace(token, "~my_account", "TYPE_XML")
print(siteExportUrl)
sleep(5)
```

Any clues on why this error and how to fix this?

Thank you.


author: kalvims
body: Hi Stan!

I got the same error and I fixed putting one more argument on the exportSpace() method. I found in this [documentation](https://developer.atlassian.com/server/confluence/remote-confluence-methods/)that the last argument is *boolean exportAll:*

```
String exportSpace(String token, String spaceKey, String exportType, boolean exportAll)
```

You can try something like this:

```
siteExportUrl = client.service.exportSpace(token, "~my_account", "TYPE_XML", True)
```

Hope this helps.


author: Vitalii Vuilov
body: Hello, guys. Could you please help to modify this script in order to export certain spaces using their keys as list instead of getting all of them from Confluence instance? Thanks.


author: Stan Ry
body: Hi [@Vitalii Vuilov](/t5/user/viewprofilepage/user-id/1213598)

Try this one.

```
'''
Simplified version of a Stephen Deutsch's script
Uses predefined list of space keys in 'spaces' variable
'''

import requests
import getpass
import shutil
from time import sleep
from requests import Session
from requests.auth import HTTPBasicAuth
from zeep import Client
from zeep.transports import Transport

user = "admin"
password = getpass.getpass()

session = Session()
session.auth = HTTPBasicAuth(user, password)
client = Client('http://localhost:1990/confluence/rpc/soap-axis/confluenceservice-v2?WSDL',
transport=Transport(session=session))

print("Logging in...")
token = client.service.login(user, password)
print("Getting Spaces")

#Specify space keys in spaces list of spaces
spaces = list()
spaces = ('myspacekey1', 'myspacekey2')
#spaces = client.service.getSpaces(token)
numSpaces = len(spaces)

for index, space in spaces:
  print("Exporting space {} of {} - {} using URL:".format(index+1, numSpaces, space))
  siteExportUrl = client.service.exportSpace(token, space, "TYPE_XML", True)
  print(siteExportUrl)
  filename = siteExportUrl.split('/')[-1]
  print("Saving as: " + filename)
  r = session.get(siteExportUrl, stream=True)
  with open(filename, 'wb') as f:
   shutil.copyfileobj(r.raw, f)
  sleep(5)
```

I've made following miniscule changes to the original script provided by Stephen Deutsch.

1. Added getpass library (import getpass adds the library namespace)

2. Added invocation of getpass() method provided by getpass library so that you don't need to specify the clear text password within the script. You'll be asked to do so at the run time.

```
password = getpass.getpass()
```

3. Added a predefined list of space keys:

```
spaces = ('myspacekey1', 'myspacekey2')
```

I am not a developer, so I've added

```
spaces = list()
```

to initialize an empty list. I don't think it's necessary here...

4. Changed enumeration so that we first scan through the list of spaces in 'spaces' with this line

```
for index, space in spaces:
```

5. Changed every use of 'spaces.key' to just 'space' because we don't have a space object anymore, we have 'space' strings taken from the 'spaces' list.

Here

```
print("Exporting space {} of {} - {} using URL:".format(index+1, numSpaces, space))
```

and here

```
siteExportUrl = client.service.exportSpace(token, space, "TYPE_XML", True)
```

6. Added the 'True' bit flag as a third parameter of the exportSpace() method's argument.

Hope this helps.

Stan


author: Vitalii Vuilov
body: Thanks a lot, Stan! I will try!


author: Vitalii Vuilov
body: After running the script I got this error:

python exportspaces.py  
File "exportspaces.py", line 37  
shutil.copyfileobj(r.raw, f)  
\^  
IndentationError: expected an indented block

P.S. fixed by shifting that block


author: Vitalii Vuilov
body: "6. Added the 'True' bit flag as a third parameter of the exportSpace() method's argument." I had to remove it otherwise it says too many arguments.


author: Stan Ry
body: [@Vitalii Vuilov](/t5/user/viewprofilepage/user-id/1213598) Thanks for edits, might help others. Seems like forum script has altered formatting, and since Python is a positional language. 'for' didn't work out. Fixed in the script.


author: Priya Pathak
body: I am getting below error could you please help me here

File "/opt/homebrew/lib/python3.11/site-packages/zeep/client.py", line 76, in __init__  
self.wsdl = Document(wsdl, self.transport, settings=self.settings)  
File "/opt/homebrew/lib/python3.11/site-packages/zeep/wsdl/wsdl.py", line 92, in __init__  
self.load(location)


author: bipin nepal
body: Hello, Any updates on this, [@Brian M Thomas](/t5/user/viewprofilepage/user-id/762083) , were you able to figure out a solution ?   

I have around 300 spaces to export from one instance (data center ) and import those spaces to another instance (data center)   

Is there any way to script both the import and export without using plugins ?   

Thinking about that many spaces, its already giving me a minor heart attack

comments: author: Bipin Nepal
body: Also looping [@Stephen Deutsch](/t5/user/viewprofilepage/user-id/781490)


author: Krishna Ponnekanti
body: Is user directory same or different ? if you have different user group set and if you have to map users to new user id , it would be problem , otherwise it would be great


author: Bipin Nepal
body: [@Krishna Ponnekanti](/t5/user/viewprofilepage/user-id/528987) in both the instances the users are stored in jira internal directory, and in both instances the email and display name is same but the username is different


author: Krishna Ponnekanti
body: how are you going to readjust or remap the old username to new username ?

If you can solve the problem, above script with minor changes will solve problem for exporting all the spaces. Importing the spaces is any how manual.


author: Bipin Nepal
body: I'm looking for a way to use the deprecated confluence [XML-RPC and SOAP APIs.](https://developer.atlassian.com/confdev/confluence-rest-api/confluence-xml-rpc-and-soap-apis)

as rest api do not have the update option :( im thinking of bulk update such usernames in the source and then do the export/ import so that in both the cases username email and display name will be the same,

I haven't found the solution yet on user update :(  

if you can help me on making changes to the above script , it would be so grateful


author: Bipin Nepal
body: [@Krishna Ponnekanti](/t5/user/viewprofilepage/user-id/528987) i have executed the avobe script and it throws  

File "src/lxml/etree.pyx", line 1848, in lxml.etree.QName.__init__  
File "src/lxml/apihelpers.pxi", line 1754, in lxml.etree._tagValidOrRaise  
ValueError: Invalid tag name u'Object\[\]'


author: Krishna Ponnekanti
body: Please use below script for exporting the spaces. My python version is 2.7.5, Based on your python version you can adjust syntax.

import os  
import subprocess  
import requests  
import json  
import time  
import datetime  
import logging  
from time import sleep

## account which is having confluence system admin access  
USER = 'username'  
## password  
PWD = 'password'

######### Bitbucket URL  
URL = '\<yourbibucketURL\>/rpc/json-rpc/confluenceservice-v2?os_authType=basic'  
begin_time = datetime.datetime.now()  
timestr = time.strftime("%Y%m%d-%H%M%S")

#now we will Create and configure logger  
logging.basicConfig(filename="export_"+timestr+".log", format='%(asctime)s %(message)s',filemode='w')

#Let us Create an object  
logger=logging.getLogger()

#Now we are going to Set the threshold of logger to DEBUG  
logger.setLevel(logging.DEBUG)

################################  
#Export Space  
################################  
def exportSpace(spaceKey):  
logger.info('Exporting Sapce')  
payload = { 'jsonrpc' : '2.0', 'method' : 'exportSpace','params' : \[spaceKey, 'TYPE_XML', 'true'\], 'id' : 7 }  
headers1 = {'Accept':'application/json', 'Content-type':'application/json' }  
resp = requests.post(URL, auth = (USER, PWD), json=payload, headers=headers1)  
logger.info(resp.status_code)  
logger.info(resp.text)  
rj = resp.json()  
logger.info(rj\["result"\])  
return rj\["result"\]

################################  
#Download Space  
################################  
def downloadSpace(downloadurl,spaceKey):  
logger.info('Download Sapce')  
OUTPUT = spaceKey+'.zip'  
nocheck = '--no-check-certificate'  
userinput = '--user='+USER  
pwdinput = '--password='+PWD  
subprocess.call(\['wget','--auth-no-challenge', userinput, pwdinput,'-O', OUTPUT, downloadurl, nocheck\])

<br />

spaces =\<get the list of space Keys \>

numSpaces = len(spaces)  
for index, space in enumerate(spaces):  
print( "space.key ----\>\>\>\>\> " +space)  
try:  
for index, spacekey in enumerate(spaces):  
print("Exporting space {} of {} - {} using URL:".format(index+1, numSpaces, spacekey))  
try:  
resu = exportSpace(spacekey)  
downloadSpace(resu,spacekey)  
except Exception as e:  
print(e)  
sleep(5)  
except Exception as ex:  
print(ex)


author: Bipin Nepal
body: Thanks, [@Krishna Ponnekanti](/t5/user/viewprofilepage/user-id/528987) i was able to export the spaces

However the problem you mentioned about user mapping is causing trouble , looks like there is no way out of this


author: Stephen Deutsch
body: Hi Brian,

What's your language of choice? Powershell? Python? Javascript?

Seeing your other question, I see two options. You could either do the request via SOAP (which is easier than you would expect), or you could script the downloading with a headless browser like Nightmare.js.

Let me know what direction you're looking to take and I'll see if I can whip something together.

comments: author: Brian M Thomas
body: I'll use whatever I can get and use... Here's what I've tried already. I began by using a Chrome browser extension called [Restlet Client](https://restlet.com/), which lets you compose requests and chain them into test sequences. I sent the series of requests described above (after first having called the **/dologin.action** ). While it has some ability to take data from previous requests (and lets the browser handle all the cookie exchanges unless you override it), it didn't have the ability to get the contents of the hidden **atl_token** field supplied by the **exportspacexml** method so as to use it in the **doexportspace** call (there were others, but their values didn't change, so I hard-coded them).

I copied the field by hand with some success, so I used its feature that translates the request into a **curl** command, which I invoked from a standard Unix shell and scripted the shell to get data from one response to use in the next. This also worked to an extent, but every time I did the **doexportspace** request, I still got permission problems.

I also tried the XMLRPC methods, as you've seen.

I have considered scripting the browser itself using Javascript, or even composing a test with Selenium, but haven't decided which way to go yet with that.

If the SOAP request is as easy as you suggest, I'd like some pointers to where I can do it more easily than I've seen; I tried the Apache CXF extension in Eclipse, and the nice, user-friendly project creation tool asked for data I had no idea how to find or compose, and required me to create my own server for testing. I gave up on that because I ran out of patience with looking up something I didn't know only to find that finding the answer required looking up something else...

I'd like most to know what nuance I'm missing in the sequence that keeps it from thinking it's coming from an authorized user. But as I said at the beginning, I'll try whatever I can get to work, and thanks for the offer.


author: Stephen Deutsch
body: If you let me know your operating system it would be helpful.


author: Brian M Thomas
body: I imagine it would; sorry for the omission in all that verbosity.

I'm using OS X 10.11.


author: Brian M Thomas
body: For the record, I'm glad that I didn't spend any more effort on trying to concoct a SOAP client, because contrary to the assertion above, it can't be done using SOAP, because though the three-parameter **doexportspace** call remains, the one taking a fourth parameter, which is a Boolean value specifying whether to do a full export, has been removed.

I am greatly piqued at this, because in all of the documentation and all of my conversations with the support people, I did not discover this until I had actually made a correct call unsuccessfully. They claim that this is documented, but saying that the interface is deprecated and *will be* removed in a future release does not inform me that *one version* of *one method* has *already* been removed, let alone forewarn me of which release will be affected *before* I waste untold hours trying to use it.


author: Brian M Thomas
body: Any hope here? I'm running well over my deadline, and even if I do script it, it will probably take days. I'd be working on it myself, but I've already spent days on that, which is why I'm behind schedule.


Tags: api

backup

export

space

xml