How I archived 100 million PDF documents... - Part 1: Reasons & Beginning
Tags: devjournal, showdev, opensource, discuss
Published: 2023-01-11 15:39:08

> **"How I archived 100 million PDF documents..." is a series about my experiences with data collecting and archival while working on the [Library of Alexandria](https://github.com/bottomless-archive-project/library-of-alexandria) project. My local instance just hit 100 million documents. It's a good time to pop a :champagne: and remember how I got here.**

## The beginning

On a Friday night, after work, most people usually watch football, go to the gym or do something useful with their life. Not everyone though. I was an exception to this rule. As an introvert, I spent the last part of my day sitting in my room, reading an utterly boring-sounding book called ["Moral letters to Lucilius"](https://en.wikipedia.org/wiki/Epistulae_Morales_ad_Lucilium). It was written by some [old dude](https://en.wikipedia.org/wiki/Seneca_the_Younger) thousands of years ago. Definitely not the most fun-sounding book for a Friday night. However, after reading it for about an hour, I realized that the title might be boring, but the contents are almost literally gold. Too bad that there were only a couple of these books that withstood the test of time.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/d0w1u18isxnb8rfejov6.jpg)
<figcaption>Good ole' Seneca<br><small>(image thanks to <a href="https://www.flickr.com/photos/18637958@N08/3314905852/in/album-72157614433871807/">Matas Petrikas</a>)</small></figcaption>

After a quick Google search, I figured out that only [less than 1% of ancient texts](https://en.wikipedia.org/wiki/Lost_literary_work) survived to the modern day. This unfortunate fact was my inspiration to start working on an ambitious web crawling and archival project, called the [Library of Alexandria](https://github.com/bottomless-archive-project/library-of-alexandria).

## But how?

At this point, I had a couple (more like a dozen) failed projects under my belt, so I was not too fond to start working on a new one. I had to motivate myself. After I set the target of saving as many documents as possible, I wanted to have a more tangible but quite hard-to-achieve goal. I set 100 million documents as my initial goal and a billion documents as my ultimate target. Ohh how naive I was.

The next day, after waking up, I immediately started typing on my old and trustworthy PC. Because I have a very T-shaped programming knowledge that is centered around **Java**, the language of choice for this project was immediately determined. Also, because I like to create small prototypes to understand the problem I need to solve, I immediately started with one.

The goal of the prototype was simple. I "just" wanted to download 10.000 documents to understand how hard it is to collect and kind of archive them. The immediate problem was that I didn't know where can I get links for this many files. [Sitemaps](https://en.wikipedia.org/wiki/Site_map) can be useful in similar scenarios. However, there are a couple of reasons why in this case they are not really a viable solution. Most of the time it doesn't contain links to the documents, or at least not to all of them. Also, I would need to get a domain list to download the sitemaps for, etc. The immediate thing that came into my mind was that it is a lot of hassle and there must be an easier way. This is when the [Common Crawl](https://commoncrawl.org/) project came into the view.

## Common crawl

Common Crawl is a project that contains hundreds of terabytes of HTML source code from websites that were [crawled](https://en.wikipedia.org/wiki/Web_crawler) by the project. They publish a new set of crawl data at the beginning of each month.

> _The crawl archive for July/August 2021 is now available! The data was crawled July 23 – August 6 and contains **3.15 billion web pages** or **360 TiB of uncompressed content**. It includes page captures of 1 billion new URLs, not visited in any of our prior crawls._

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/36ldkuagubse1kpcw421.PNG)
<figcaption>Tiny little datasets...</figcaption>

It sounded exactly like the data that I needed. There was just one thing left to do. Grab the files and parse them with an HTML parser. This was the time when I realized that no matter what I do, it's not going to be an easy ride. When I downloaded the first entry provided by the Common Crawl project, I noticed that it was saved in a strange file format called [WARC](https://en.wikipedia.org/wiki/Web_ARChive).

I found one Java library on [Github](https://github.com/Mixnode/mixnode-warcreader-java) (thanks Mixnode) that was able to read these files. Unfortunately, it was not maintained for the past couple of years. I picked it up and [forked it](https://github.com/laxika/java-warc) to make it a little easier to use. (A couple of years later this repo was [moved under](https://github.com/bottomless-archive-project/java-warc) the Bottomless Archive project as well.)

Finally, at this point, I was able to go through a bunch of webpages (parsing them in the process with [JSoup](https://jsoup.org/)), grab all the links that contained pdf files based on the file extension then download them. Unsurprisingly, most of the pages (~60-80%) ended up being unavailable ([404 Not Found](https://en.wikipedia.org/wiki/HTTP_404#Soft_404) and friends). After a quick cup of coffee, I got the 10.000 documents on my hard drive. This is when I realized that I have one more problem to solve.

## Unboxing & validation

So, when I started to view the documents, a lot of them simply failed to open. I had to look around for a library that could verify PDF documents. I had some experience with [PDFBox](https://pdfbox.apache.org/) in the past, so it seemed to be a good go-to solution. It had no way to verify documents by default, but it could open and parse them and that was enough to filter out the incorrect ones. It felt a little bit strange just to read the whole PDF into the memory to verify if it is correct or not, but hey I needed a simple fix for now and it worked really well.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/kpog69leh9t2kslgafj6.PNG)
<figcaption>Literally, half of the internet.</figcaption>

After doing a re-run, I concluded that 10.000 perfectly valid documents can fit on around 1.5 GB of space. That's not too bad I thought. Let's crawl more because it sounds like a lot of fun. I left my PC there for about half an hour, just to test the app a bit more.

> **[Part two](https://dev.to/laxika/how-i-archived-100-million-pdf-documents-part-2-ffg) will describe how more challenges were solved like parallelizing the download requests, splitting up the application, making the documents searchable, and adding a web user interface.**