Marco De Lellis

Nice article!

I also did something similar, and while it was quite rough, it got the job done.

An old PDF book had some really bad scanning and was painful to read: the idea was simple, convert one page into an image, feed it to a model, convert into markdown, append to the already processed text and repeat.

Among the many ways to convert the PDF page into an image, these bash commands worked really well in order to extract a particular page:

export n=4    #  page number
pdftk ~/mybook.pdf cat $n output /tmp/$n.pdf    # extract page 4
convert -density 300 -trim /tmp/$n.pdf -quality 100 /tmp/$n.jpg    # convert page 4 into JPEG
rm /tmp/$(($n-1)).{pdf,jpg}    # remove page 3 files 

The model that seemed best suited for the job, OvisOCR2, has been loaded by llama-sever, listening on http://127.0.0.1:8080:

llama-server -hf Abiray/OvisOCR2-GGUF:Q8_0 --no-webui

Every JPEG image, encoded in base64 and embedded into a JSON template, was sent as an attachment (@/path/to/file) to llama-server by the following curl command, where API_URL="http://127.0.0.1:8080/v1/chat/completions", IMG_PATH=/tmp/$page.jpg, and the response was saved as a json.md file, an extension that hints to the Markdown content embedded in the JSON response:

curl -s $API_URL -d @/tmp/$page.json > /tmp/$page.json.md

while the base64 encoding and JSON template were obtained by emitting the template with cat, and using envsubst to substitute $IMG_BASE64 in the template:

IMG_BASE64=$(base64 -w 0 "$IMG_PATH")
cat > /tmp/$page.json << EOF | envsubst
{
  "messages": [
    {
      "role": "user",
      "content": [
        {
          "type": "text",
          "text": "<|im_start|>user\nExtract all readable content from the image in natural human reading order and output the result as a single Markdown document. Format formulas as LaTeX. Format tables as HTML: <table>...</table>. Preserve the original text without translation.<|im_end|>\n<|im_start|>assistant\n"
        },
        {
          "type": "image_url",
          "image_url": {
            "url": "data:image/jpeg;base64,$IMG_BASE64"
          }
        }
      ]
    }
  ],
  "max_tokens": 4096,
  "temperature": 0,
  "top_p": 1,
  "stream": false
}
EOF

The powerful jq was used to extract the Markdown from the content element in JSON.

cat /tmp/$page.json.md  |  jq --raw-output '.choices[0].message.content' >> "$md_file"

You can easily assemble those comands into a full bash script that takes the page number as a parameter, and run int inside a for loop: if the book has 300 pages, you can get a full Markdown version by running the script inside a loop like this one:

for n in $(seq 1 300) ; do ./convert.sh $n ; done

Hope that it will be useful .