How Should I get all text from all spaces and their child content?
Asked by Humashankar VJ:

Hello - Looking for some insight on this problem statement:

How to retrieve all content from all global spaces in your Confluence instance

**Problem Statement:**

I want to get all the content from all spaces with type "global".  

This now costs more than 70 API calls. This is because there is a limit of 25 objects per call and I need to get the "space key" from every space manually to get their content via a for loop with calls to "[https://wiki-acc.\[mycompany\].com/rest/api/space/\[space_key\]/content?expand=body.storage](https://wiki-acc.wearetriple.com/rest/api/space/AJAX/content?expand=body.storage)".  

If there is a way to limit the amount of API calls to achieve my goal, I would very much like to know. It now takes about 37 seconds to execute, which is too slow for my use case.  

Thanks in advance!

Reference ticker and latest code:

[How can I get all text from all spaces and their c... (atlassian.com)](https://community.atlassian.com/t5/Confluence-questions/How-can-I-get-all-text-from-all-spaces-and-their-child-content/qaq-p/2616288#M294133)

CC: [@Bas Hulskamp](/t5/user/viewprofilepage/user-id/5435175)

Warm Regards


Answers:
author: Bas Hulskamp
body: Hi [@Humashankar VJ](/t5/user/viewprofilepage/user-id/5409474) [@Aron Gombas _Midori_](/t5/user/viewprofilepage/user-id/502447) [@Barbara Szczesniak](/t5/user/viewprofilepage/user-id/5187859)   

Thank you all for your assistance!  

I have solved my issue now, by using async functions to retrieve the data with the Confluence API. This took the execution time down to \<=3 seconds from the +/-60 seconds that it was before.  

This is my code

```
async def get_all_global_space_keys(self) -> list[str]:

        url = self.base_url + "/rest/api/space"

        params = {"type": "global"}

        all_keys = []

        async with aiohttp.ClientSession() as session:

            while True:

                async with session.get(url, headers=self.headers, params=params) as response:

                    data = await response.json()

                    for result in data["results"]:

                        all_keys.append(result["key"])

                    next_url = data['_links'].get('next', None)

                    if next_url:

                        url = self.base_url + next_url

                    else:

                        break

        return all_keys

    

    async def get_content_from_space(self, session, space_key) -> list[Document]:

        documents = []

        url = self.base_url + "/rest/api/space/" + space_key + "/content"

        params = {"expand": "body.storage"}

        async with session.get(url, headers=self.headers, params=params) as response:

            data = await response.json()

            for result in data["page"]["results"]:

                content_url = self.base_url + result["_links"]["webui"]

                # content: str = BeautifulSoup(result["body"]["storage"]["value"], 'html.parser').get_text()

                content: str = result["body"]["storage"]["value"]

                documents.append(Document(content, metadata={"source": content_url}).to_json())

        return documents

    async def get_all_content_from_all_global_spaces(self, space_keys: list[str]):

        documents: list[Document] = []

        async with aiohttp.ClientSession() as session:

            tasks = [self.get_content_from_space(session, key) for key in space_keys]

            results = await asyncio.gather(*tasks)

            for result in results:

                documents.extend(result)

        with open(self.WIKI_DATA_PATH, 'w', encoding='utf-8') as file:

            file.write(json.dumps(documents))

    async def get_all_async(self):

        space_keys = await self.get_all_global_space_keys()

        await self.get_all_content_from_all_global_spaces(space_keys)

c = ConfluenceService()
asyncio.run(c.get_all_async())
```

comments: author: Humashankar VJ
body: Great to know [@Bas Hulskamp](/t5/user/viewprofilepage/user-id/5435175) - Have a great one, let me mark this as solved for other reference in the future


author: Humashankar VJ
body: Hi [@Barbara Szczesniak](/t5/user/viewprofilepage/user-id/5187859)

Can you take a look in this use case and help [@Bas Hulskamp](/t5/user/viewprofilepage/user-id/5435175)

Regards

comments: author: Barbara Szczesniak
body: The reference question is tagged with Data Center, but this question is tagged for Cloud.

I work with Cloud, and I do not use Rest API, so I'm not sure I can provide assistance.

My first questions are:

* What are you using the content for---what is the reason you need all of the content? This may impact the format you the content in.
* Why not export the spaces to the format you need?


author: Humashankar VJ
body: Hi [@Bas Hulskamp](/t5/user/viewprofilepage/user-id/5435175) - Kindly take the questions for Barbara


author: Bas Hulskamp
body: Good afternoon [@Barbara Szczesniak](/t5/user/viewprofilepage/user-id/5187859) ,  

I use the REST API tot query an on premise instance of Confluence.  

All the text of pages and files I use as content for my RAG (Retrieval Augmented Generation) application. RAG is an AI principle if you didn't know.  

It now takes 74 seconds to get all content of all spaces (as explained in the linked post), because I have to call the API at least an amount of times equal to the amount of spaces to get all the content with this endpoint "[https://wiki-acc.\[mycompany\].com/rest/api/space/\[space_key\]/content?expand=body.storage](https://wiki-acc.wearetriple.com/rest/api/space/AJAX/content?expand=body.storage)". Not to mention having to iterate over the spaces to get the SpaceKeys, with a limit of 25 per call because I couldn't get the total amount of spaces from an endpoint (see code in original post).  

I would like to know if there is an alternative way that takes less API calls and is faster.  

I'm not sure what the "export spaces" feature is that you mentioned... how does that work? Would that help me in this case?


author: Bas Hulskamp
body: [@Barbara Szczesniak](/t5/user/viewprofilepage/user-id/5187859) Could you please answer my question? Letting me know if you don't know the answer would also be great


author: Barbara Szczesniak
body: [@Bas Hulskamp](/t5/user/viewprofilepage/user-id/5435175) I don't know anything about API calls or on premise Confluence.

According to this page (<https://confluence.atlassian.com/doc/export-content-to-word-pdf-html-and-xml-139475.html> ), you can export pages or a space to Word, HTML, PDF, or XML formats. I just thought maybe you could use this output for whatever you were using the content for---your RAG application, I guess. I'm not sure if this format would suit your needs.


author: Bas Hulskamp
body: [@Barbara Szczesniak](/t5/user/viewprofilepage/user-id/5187859) I don't think so... I would have to read the files and then process the text... directly having the HTML strings is faster en more effective I think.  

Do you know of someone else that could help me? Or is my request not possible with the current API? Note: I use the V1 API, because the Confluence instance isn't on the latest version (and I'm not the administrator).


author: Barbara Szczesniak
body: Your question is visible to everyone in the community, so someone else might see it and answer.


author: Humashankar VJ
body: Hi [@Barbara Szczesniak](/t5/user/viewprofilepage/user-id/5187859) - Thanks for taking your time on this use case.

Hi [@Bas Hulskamp](/t5/user/viewprofilepage/user-id/5435175) - Let me also find some one who is having exposure on this as this is aa unique problem statement.


author: Humashankar VJ
body: Hi [@Aron Gombas _Midori_](/t5/user/viewprofilepage/user-id/502447) - Can you take your time and assist whether you able to assist on this topic / route to the right team who can help on this ?


author: Bas Hulskamp
body: [@Humashankar VJ](/t5/user/viewprofilepage/user-id/5409474) Thanks very much for your assistance!   
I'm looking foward to hearing from someone soon.


author: Aron Gombas _Midori_
body: There are at least two opportunities for parallelization:

1. When you get the first page of the spaces, you can immediately start collecting the contents in those spaces. In other words, you don't need to load all spaces before you can start collecting the pages in those.
2. You can, obviously, collect the pages in two separate spaces in parallel.

Using these two trivial tricks, you can massively accelerate the data collection here.

Note that you may run into rate limits if you go over-aggressive with parallelization. So it is a bit of experimentation and tuning.

comments: author: Humashankar VJ
body: Thanks [@Aron Gombas _Midori_](/t5/user/viewprofilepage/user-id/502447) for the insight

Hi [@Bas Hulskamp](/t5/user/viewprofilepage/user-id/5435175) - Can you intake the above data points and see how you want to move on ?


author: Bas Hulskamp
body: Hi [@Humashankar VJ](/t5/user/viewprofilepage/user-id/5409474)   

I'll see what I can do with this. Thanks for your insight [@Aron Gombas _Midori_](/t5/user/viewprofilepage/user-id/502447) !  
Seems a bit like an AI answer by the way haha


author: Aron Gombas _Midori_
body: ?


author: Bas Hulskamp
body: Anyone?

comments: author: Bas Hulskamp
body: [@Humashankar VJ](/t5/user/viewprofilepage/user-id/5409474) I'm not familiar with the people who are active on this forum. Can you perhaps @ mention someone that would know more about my question?


author: Humashankar VJ
body: Sure [@Bas Hulskamp](/t5/user/viewprofilepage/user-id/5435175) - Let me work on that part.


Tags: cloud

confluence-cloud

data-center