Creating a searchable knowledge-base from several PDF files
Tags: knowledgebase, bash, combinepdf, websitefrompdf
Published: 2022-03-01 12:11:46

---
title: Creating a searchable knowledge-base from several PDF files
published: true
description: Using bash and other tools to process pdfs files to create a searchable knowledge-base for all Sabbath School Lessons since 1888
tags: knowledgebase, bash, combinepdf, websitefrompdf
//cover_image: https://direct_url_to_image.jpg
---

## Part 1 - Exploring the Power of Bash
In the current project we are going to work with several isolated PDF files, converting them into a searchable knowledge base. We are going to see how to:
1. **Download** a file from a website using a script.
2. **Scrape** an html table for links.
2. **Batch download** several files in parallel without running out of memory.
3. **Combine pdf files** using a script
3. **Batch rename files**
4. Create a website from pdf files.

Before we begin, let us give a little background for the project. We have quarterly Bible study guides that have faithfully been produced every year since 1886. All of these have been digitized and [are available for all years since 1888](https://www.adventistarchives.org/sabbathschoollessons) (also available [here](https://documents.adventistarchives.org/SSQ/Forms/AllItems.aspx)). Like in all areas, the faithful student is he that compares the current studies with what has been done in the past. This comparison will help in understanding the subject under consideration more deeply and in seeing if there are any errors that have been introduced in the new, or if there were errors in the old. And Christ said that "every scribe instructed concerning the kingdom of heaven is like a householder who ***brings out of his treasure things new and old***."

To be able to make use of the old guides, the student currently has to go to the index of titles and check for the guides relevant to his subject of study, download all the pdf files individually, the reading each of them. But it would be more desirable to have a searchable knowledge-base having all these guides which the student can search whatever he needs at once, without having to manage 500+ individual files. It is such a system that we will be building.

In the first step we are going to:

## 1. Download a pdf file from a website using a script
The first lesson available is found at the following link: `http://documents.adventistarchives.org/SSQ/SS18880101-01.pdf`

Downloading this single file is pretty straightforward.

```bash
wget http://documents.adventistarchives.org/SSQ/SS18880101-01.pdf
```
But since there are over 500 files and we cannot possibly copy and paste the link one by one from the site and download them, we are going to:

## 2. Scape the page for links to pdf files
All the pdf files we need are available in the directory being served at `http://documents.adventistarchives.org/SSQ/`. But trying to access that directory does not give a listing of the files in it, but rather redirects to a different page. We would use `recursive wget` to download all those files if accessing the directory gave a listing of the files it contains. But since this is not available to us, we have to use some other method.

Since we already know that all the files we are looking for have `http://documents.adventistarchives.org/SSQ/` in their url. That is going to form the basis for our pattern. The final command that we have to extract both the url of the pdf file and the lesson title for that pdf is

```bash
echo -n '' > files.txt && curl -s https://www.adventistarchives.org/sabbathschoollessons | grep  -o 'http://documents.adventistarchives.org/SSQ/[^.]*.pdf\"[^\>]*[^\<]*' | while read line ; do echo "$line" | sed -e 's/\"[^\>]*./ /' | sed -e 's/  \+/ /g' >> files.txt; done
```
`grep -o` will print only the matches found.

The result is saved in `files.txt`. Each line has the pdf url and the lesson title for that pdf.
![files.txt](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/fkbph31vfs8yn4h508h3.png)

To see the number of files that we will be dealing with we can do `wc -l files.txt`. There are 516 files in total.

## 3. Downloading all 510 files
The files are in reality a little fewer than 516 because the list contains duplicates since it is designed for every quarter whereas some pdf files have two quarters. To remove the duplicated urls, we use sort:
```bash
sort -u -o files.txt files.txt
```
To download the files, we need to get the url for each file from `files.txt`. We can do this by
```bash
cat files.txt |while read line ; do echo "$line" | cut -f 1 -d " " |xargs wget; done
```

But this downloads one file at a time and is as a result very slow and inefficient.

To download the files in parallel, we use wget with the '-b' option so that it can fork itself. We sort the urls file to remove duplicated urls since files.txt has duplicated urls and not duplicated lines because of different lesson titles for the duplicated urls.

```bash
cut -d\  -f1  files.txt > urls && sort -u -o urls urls && cat urls |while read line ; do echo "$line" | cut -f 1 -d " " |xargs wget -b; done
```

We are probably very lucky to download all the 510 files in 5 seconds without running out of 1GB of memory. It would be interesting to know how this would perform in node.

## 4. Combine 510 pdfs into 1
The first obvious step in making our knowledge-base is probably combining all the pdfs into one. This will result in a single big pdf (over 100mbs in size), but much more useful in study if it can be opened than 510 separate pdfs. For this we are going to use `ghostscript`:
```bash
gs -dBATCH -dNOPAUSE -q -sDEVICE=pdfwrite -sOutputFile=allLessons.pdf *.pdf
```

Ghostscript quickly runs us out of memory. A better alternative for this task could be `pdfunite`.

```bash
pdfunite *.pdf allLessons.pdf
```
`pdfunite` gets us a bigger file, but it is also runs us out of memory. Using NitroPro on windows in a system having 8GB of RAM gives us a file of 708.4Mb. It would be interesting to see if this would also be possible in a single core 1GB RAM system like the linux one we are trying to force to work for us. 

Let's try pdfunite on 10 files at a time. We will use `GNU parallel` and let it do what it can to manage the memory for us. This process takes about ***20 minutes*** and outputs files 1.pdf ... 51.pdf which we need to further combine.

```bash
ls -lha ./ |grep SS.*pdf | grep -o '[^ ]*$' |parallel -N10 pdfunite {} {#}.pdf
```

## 6. Creating a site where pdfs can be downloaded.
We are going to do this using github pages. To do this we are first going to create a CNAME entry in our DNS server to point to github pages.

|CNAME|ttl|points to
|-----|---|------
|sslpdfs|900|gospelsounders.github.io.

Then we are going to add a CNAME file with the contents `sslpdfs.gospelsounders.org`, commit and push repo to github, and configure the github pages settings in the github repo. 

```bash
GIT_SSH_COMMAND='ssh -i /tmp/gitKey -o IdentitiesOnly=yes' git push origin master
```
Next we should create the index of the files that we have and put them in README.md. The following bash commands do this for us.

```bash
curl -s https://www.adventistarchives.org/sabbathschoollessons |grep -o ^[\<]td.* |sed -e 's/<td[^\>]*>//g' | sed -e 's/<sup>.*<\/sup>//g' | sed -e 's/<\/td>//g' |sed -e 's/\<a.*href="[^h][^t][^t][^p][^:][^>]*>.*<\/a><//' | sed -e 's/<a[^>]*><\/a>//g' |sed -e 's/<[^a][^>]*>//g' |grep -A 2 '^[0-9]\{4\}' | grep -v -- "^--$" |parallel -N3 echo {} | sed -e 's/\(<a[^<]*\)\(<.*\)/\1/g'| sed -e 's/<a.* href="\([^"]*\)"[^>]*>\(.*\)/[\2](\1)/' | sed -e 's/  */ /g' | sed -e 's/http:\/\/documents.adventistarchives.org\/SSQ\///' | sed -e 's/ \]/\]/g'
```

It produces a list in the following format:

![IndexofTitles](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/zx563v6du2bw5zvohhi4.png)

But we would desire to have all the quarters of a year in a single row in a table having the following header:


| Year | Quarter 1 | Quarter 2 | Quarter 3 | Quarter 4|
|------|-----------|-----------|-----------|----------|

With the header already in the markdown file (README.md), the following commands create the table for us and add it to the markdown file.
```bash
curl -s https://www.adventistarchives.org/sabbathschoollessons |grep -o ^[\<]td.* |sed -e 's/<td[^\>]*>//g' | sed -e 's/<sup>.*<\/sup>//g' | sed -e 's/<\/td>//g' |sed -e 's/\<a.*href="[^h][^t][^t][^p][^:][^>]*>.*<\/a><//' | sed -e 's/<a[^>]*><\/a>//g' |sed -e 's/<[^a][^>]*>//g' |grep -A 2 '^[0-9]\{4\}' | grep -v -- "^--$" |parallel -N3 echo {} | sed -e 's/\(<a[^<]*\)\(<.*\)/\1/g'| sed -e 's/<a.* href="\([^"]*\)"[^>]*>\(.*\)/[\2](\1)/' | sed -e 's/  */ /g' | sed -e 's/http:\/\/documents.adventistarchives.org\/SSQ\///' | sed -e 's/ \]/\]/g' | parallel -N4 echo {} | sed -e 's/ [0-9]\{4\} [1-4][a-z]\{2\} / \|/g' |sed -e 's/ 1st /\| /' >> README.md 
```
## 7. Continue combining the 51 files
We first have to add a leading zero to the first 9 files, then run `pdfunite`
```bash
rename.ul "" 0 ?.pdf

ls -lha ./ |grep " [0-9][0-9]\.pdf" | grep -o '[^ ]*$' |parallel -N10 pdfunite {} {#}-2.pdf

ls -lha ./ |grep " [0-9]-2\.pdf" | grep -o '[^ ]*$' |parallel -N6 pdfunite {} allLessons.pdf
```

Last process still breaks because of limited memory. So we will upload the file that we created using NitroPro to our server.

```bash
scp All\ Lessons.pdf user@server:/tmp/pdfs/allLessons.pdf
```

`allLessons.pdf` is greater than 100MBs (708.4MB) and we can't push to github. We need to find a way of reducing its size. So let's try to remove the images from the pdfs and see the result.

```bash
gs -sDEVICE=pdfwrite -dFILTERVECTOR -dFILTERIMAGE -o allLessonsSmall.pdf "allLessons.pdf"
```

We are almost done as this gives us a file that is 68.5MBs. But the ~~text is white on a white background~~the text is transparent.

Using `Evince`, the default pdf viewer for Ubuntu, the transparent-text pdf can be read by selecting all the text using `CTRL+A` which will make the text visible. But this method seems not to work on all pdf viewers including 'Foxit Reader' and Adobe pdf readers. 
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/kogloobu1f8xu5pmig1r.png)
We will therefore convert the transparent-text pdf to html and change the color in the generated css file.

For this exercise we have chosen to use a digital ocean trial account to launch a 4 Core 8GB RAM 160 GB SSD server. The following command is executed unbelievably fast (2m8.701s):

```bash
time pdf2htmlEX --split-pages 1 --process-nontext 0 --process-outline 0 --process-annotation 0 --process-form 0 --embed-css 0 --embed-font 0 --embed-image 0 --embed-javascript 0 --embed-outline 0 --dest-dir html allLessonsSmall.pdf
```

The result is 26672 files of total size 223MB. You can find these by running `ls html |wc -l` and `du -h html`. All text is still set to be transparent as seen from the following css file:
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/l79ghnbi1lnc67loljcv.png)
This is changed to black as follows:
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/6r9hyiz3smbchv5vdqjw.png)


## 7. Converting all the pdfs to html

We will now convert the pdfs to html files, which will allow us to be able to search their contents within github. This is also an easy task using our free trial 4 core server:

```bash
ls -lha ./ |grep SS.*pdf | grep -o '[^ ]*$' |parallel -N10 echo {} | sed 's/ /\n/g' | parallel -N1 pdf2htmlEX --split-pages 1 --dest-dir htmls {}
```

## 8. Adding to git, commiting and pushing to github

For some reason, these command add all the files at once.
```bash
ls ./|head -n $((1*18135/5))|tail -n $((18135/5))|xargs git add```

```bash
for file in $(ls ./|head -n $((1*18135/5))|tail -n $((18135/5)) ); do git add $file ; done
```

So we move the htmls folder out of our folder, recreate it  copy back the files in batches, add the files to git and commit. We run this command 5 times, incrementing the multiplier (1 in this command).

```bash
for file in $(ls ../htmls|head -n $((1*18135/5))|tail -n $((18135/5)) ); do cp "../htmls/$file" htmls/ ; done && git add . && git commit -m "added batch of files"
```

## 9. Editing the index to add links to html files
```bash
sed -i '/SS.*pdf/s/(\([^)]*\))/(htmls\/\1.html) \\| [⇩](\1)/g' README.md
```

## 10. Converting all the pdfs to text
In docs folder,
```bash
ls -lha ./ |grep SS.*pdf | grep -o '[^ ]*$' |parallel -N10 echo {} | sed 's/ /\n/g' | parallel -N1 pdftotext -layout {}

ls -lha ./ |grep "[0-9].txt" | grep -o '[^ ]*$' |parallel -N10 echo {} | sed 's/ /\n/g' | parallel -N1 mv {} texts/
```

## 11. Remove transparency from htmls
The htmls contain transparent text which we wish to convert to black. In `docs/htmls` run:

```bash
sed -i '/fc0{color:transparent;/s/transparent/black/' *.html
sed -i '/fc0{color:transparent;/s/transparent/black/' *.css
```

## 12. Add text files to index
```bash
sed -i '/SS.*pdf/s/(\([^(]*\).pdf)/(\1.pdf) \\| [txt](texts\/\1.txt)/g' README.md
```