insights-extractor 0.1.6
Efficient PDF analysis, text extraction, preprocessing, and pattern recognition with customizable configurations and utilities.
Requires Python: >=3.7
Keywords: nltk,spaCy,Pymupdf,Camelot,OpenCV,Ghostscript,insight-extractor,pdf extraction,pdf data extraction,pdf data,classification,keyword extraction

# Extractlib

This is a Python package that provides a set of tools and utilities for processing and analyzing PDF documents. It includes functionality for extracting text and tables from PDFs, cleaning and preprocessing text data, and analyzing content for keywords and patterns. The package also provides a number of configuration options for customizing the behavior of the tools and utilities, making it flexible and easy to use in a variety of different contexts. Whether you need to extract data from PDF documents for data analysis, or analyze PDF content for specific keywords or patterns, this package provides the tools you need to get the job done quickly and efficiently.


## Dependency Overview
### Python Dependency Install
This project leverages packages 
<pre>
nltk == 3.8.1 
spacy == 3.5.2
PyMuPDF == 1.21.1 
camelot-py == 0.11.0 
opencv-python == 4.7.0.72 
ghostscript == 0.7
</pre>

## manually install supporting binaries

### spacy dependencies
<pre>python -m spacy download en_core_web_sm</pre>

### camelot dependencies
- https://camelot-py.readthedocs.io/en/master/user/install-deps.html#install-deps

#### Ubuntu
<pre>$ apt install ghostscript python3-tk</pre>

#### MacOS
<pre>$ brew install ghostscript tcl-tk</pre>

## windows dependency installations
- Install ghostscript: https://ghostscript.com/releases/gsdnld.html
- Install Tinker: https://platform.activestate.com/activestate/activetcl-8.6/auto-fork?_ga=2.93217438.2024444162.1679060315-1994225326.1678735799
 
# Configuration
- configuration file should be placed in the root of the project and named 'extractlib.config.json'

## Example 'extractlib.config.json' File
### this file should be located in the root of your project
<pre>
{
  "std_out_logging": true,
  "supported_file_types": [".pdf"],
  "invalid_content_regexs": ["X{2,}"],
  "stop_words": [
    "na",
    "dependent",
    "address",
    "plans",
    "network",
    "nonnetwork",
    "additional",
    "covered"
  ],
  "keywords": {
    "dental": 5,
    "vision": 5,    
    "life": 5,
    "disability": 5
  },
  "keyword_synonyms": {
    "dental": ["orthodontic", "Endo", "Perio", "Oral"],
    "vision": ["eye", "vision", "lens", "lenses", "contact", "contacts"],
    "life": [
      "accident",
      "critical",
      "illness",
      "accidental",
      "dismember",
      "AD&D"
    ],
    "disability": []
  },
  "word_min_length": 3
}
</pre>

# access config variables
<pre>
from extractlib.settings import config

print(json.dump(config.config_raw, indent=4))
</pre>
# Example implementation
<pre>
from extractlib.document.process import process_document
import json

def main(file: str):
    result = process_document(file,  exclude_pages=[2,3], use_multithreading=False, split_pages_output_dir='./output', delete_split_pages=False)
    # Save the HTML content to a temporary file
    with open('temp.json', 'w') as f:
        json.dump(result, f, indent=4)


if __name__ == '__main__':

    # get working directory
    import os
    target_dir = os.path.dirname(os.path.abspath(__file__))
    main(f'{target_dir}/_testdata/PDF.pdf')
</pre>