litdb 2.2.1
A literature database tool with GPT integration.

#+title: litdb - a literature and document database

#+attr_org: :width 600
[[./litdb.png]]
#+BEGIN_html
<a href="https://github.com/jkitchin/litdb/actions/workflows/build.yml">
  <img src="https://github.com/jkitchin/litdb/actions/workflows/build.yml/badge.svg" alt="Build Status">
</a>
<img src="https://img.shields.io/badge/tests-133_passed,_66_skipped-brightgreen" alt="Tests">
#+END_HTML

* litdb concept

litdb is a tool to help you curate and use your collection of scientific literature. You use it to collect and search papers. You can use it to collect older articles, and to keep up with newer articles. litdb uses https://openalex.org for searching the scientific literature, and https://turso.tech/libsql to store results in a local database.

The idea is you add papers to your database, and then you can search it with natural language queries, and interact with it via an ollama GPT application. It will show you the papers that best match your query. You can read those papers, get bibtex entries for them, or add new papers based on the references, papers that cite that paper, or related papers. You can also set up filters that you update when you want to get new papers created since the last time you checked.

** videos

1. https://www.youtube.com/live/e-J3Bh2Uti4 Introduction to litdb
2. https://www.youtube.com/live/teW68WogulU local files (volume is very low for some reason)
3. https://youtube.com/live/3LltpiiQaR8 CrossRef, reviewer suggestions, COA
4. https://youtube.com/live/ZkKKuvVUWkE litdb and Emacs
5. https://youtube.com/live/j7rItPwWDaY litdb and Jupyter Lab
6. https://youtube.com/live/SUtvtc7l6y0 litdb + GPT enhancements
7. https://youtube.com/live/3FZ1ROnCC6Y litdb + LiteLLM and streamlit
8. https://www.youtube.com/live/IKKTQSTXQmc litdb + Youtube and audio
9. https://youtube.com/live/MEf9rPI0Z1M  litdb + Image search with text and image queries using CLIP
10. https://youtube.com/live/C4qCam0shf8  litdb + deep research
11. https://youtube.com/live/6Wpy7KM3wIM  litdb + LLM-augmented search of OpenAlex
12. https://www.youtube.com/live/eS0D-Aje_6A litdb + Claude Desktop

** installation

litdb is on PyPi.

#+BEGIN_SRC sh
pip install litdb
#+END_SRC


To get the cutting edge package, you can install it directly from GitHUB.

#+BEGIN_SRC sh
pip install git+https://github.com/jkitchin/litdb
#+END_SRC

litdb relies on a lot of ML-related packages, and conflicts with version are common. If you have any issues, I recommend you install it with uv (https://github.com/astral-sh/uv) like this:

#+BEGIN_SRC sh
uv venv --python 3.13
source .venv/bin/activate
uv pip install litdb
#+END_SRC

You have to activate that virtual env to use litdb, which may be annoying.

** Recent additions

*** Version 2.1.8

- *fromtext command*: Extract and add references from pasted text using LLM parsing and CrossRef matching. See [[./FROMTEXT_USAGE.md][detailed documentation]].
- *summary command*: Generate newsletter-style summaries of recent articles with automatic topic extraction and classification.
- *extract command*: Extract tables from PDF files.
- *schema command*: Extract structured data from documents using a flexible schema DSL or JSON format.
- *Bug fix*: Corrected OpenAlex API parameter from ~email~ to ~mailto~ for proper API compliance.
- *Security fix*: Replaced unsafe ~eval()~ with ~ast.literal_eval()~ in schema parsing.

** configuration

You have to create a toml configuration file. This file is called litdb.toml. The directory this file is in is considered the root directory. All commands will start in the current working directory and look up to find this file. You can put this file in your home directory, or you can have sub-directories, e.g. a per project litdb.

There are a few choices you have to make. You have to choose a SentenceTransformer model, and specify the size of the vectors it makes. You also have to specify the chunk_size and chunk_overlap settings that are used to break documents up to compute document level embedding vectors.

You will need an OpenAlex premium key if you want to use the update-filters feature.

#+BEGIN_EXAMPLE
[embedding]
# SentenceTransformer model for vector search
model = 'all-MiniLM-L6-v2'
cross-encoder = 'cross-encoder/ms-marco-MiniLM-L-6-v2'
chunk_size = 1000
chunk_overlap = 200

[openalex]
# Email for OpenAlex API (polite pool access)
email = "you@example.com"
# Optional: Premium API key for update-filters feature
api_key = "..."

[gpt]
# Model for the 'litdb gpt' command (uses ollama)
model = "llama2"

[llm]
# Model for LiteLLM-based commands (chat, fromtext, summary, schema, research)
# Supports any provider: ollama/*, openai/*, anthropic/*, google_genai/*, etc.
model = "ollama/llama2"
#+END_EXAMPLE

Configuration sections and their usage:
- *[embedding]*: Used by all vector search operations (~vsearch~, ~hybrid-search~, etc.)
- *[openalex]*: Required for OpenAlex searches and updates
- *[gpt]*: Used by the ~litdb gpt~ command (ollama-only)
- *[llm]*: Used by ~chat~, ~fromtext~, ~summary~, ~schema~, and ~research~ commands (supports all LiteLLM providers)

You can define an environment variable to the root of your default litdb project. This should be a directory with a litdb.toml file in it.

#+BEGIN_SRC sh
export LITDB_ROOT="/path/to/your/default/litdb"
#+END_SRC

When you run a litdb command, it will look for a dominating litdb.toml file, which means you are running the command in a litdb project. If one is not found, it will check for the LITDB_ROOT environment variable and use that if it is found. Finally, if that does not exist, it will prompt you to make a new project in the current directory.

* Using litdb

Your litdb starts out empty. You have to add articles that are relevant to you. It is an open question of the best way to build a litdb. The answer surely depends on what your aim is. You have to compromise on breadth and depth with the database size. The CLI makes it pretty easy to do this

litdb has a cli with an entry command of litdb and subcommands (like git) for interacting with it. You can see all the options with this command.

#+BEGIN_SRC sh :dir example
litdb --help
#+END_SRC

** Command Organization

litdb's 43 commands are organized into logical groups:

- *Management*: init, add, remove, index, reindex, update_embeddings
- *Search*: vsearch, fulltext, hybrid_search, lsearch, similar, image_search, screenshot
- *Export*: bibtex, citation, show, visit, about, sql
- *Tags*: add_tag, rm_tag, delete_tag, show_tag, list_tags
- *Review*: review, summary
- *Filters*: add_filter, rm_filter, update_filters, list_filters
- *OpenAlex*: openalex, author_search, follow, watch, citing, related, unpaywall
- *Research*: fhresearch, research, suggest_reviewers
- *Data Processing*: crossref, fromtext, extract, schema, crawl
- *Utilities*: web, audio, chat, app, version, coa

** Searching the web

You have to start somewhere. You can use this to open a search in OpenAlex.

#+BEGIN_SRC sh
litdb web query
#+END_SRC

You can also open searches with these options:

| option                | source         |
|-----------------------+----------------|
| -g, --google          | Google         |
| -gs, --google-scholar | Google Scholar |
| -ar, --arxiv          | Arxiv          |
| -pm, --pubmed         | Pubmed         |
| -cr, --chemrxiv       | ChemRxiv       |
| -br, --biorxiv        | BioRxiv        |
| -a, --all             | All            |

You can find starting points this way.

*** Fine-tuned search in OpenAlex

This is a default query in Open Alex. It does not change your litdb, it just does a simple text search query on works.

#+BEGIN_SRC sh
litdb openalex query
#+END_SRC

You can get more specific with a filter:

#+BEGIN_SRC sh
litdb openalex -f 'author.orcid:https://orcid.org/0000-0003-2625-9232'
#+END_SRC

You can also search other endpoints and use fulters. Here we perform a search on Sources for display_names that contain the word discovery.

#+BEGIN_SRC sh
litdb openalex -e sources -f display_name.search:discovery
#+END_SRC

*** LLM-enhanced search of OpenAlex

The previous command lets you define a precise keyword or filter search for OpenAlex. An alternative is the ~lsearch~ sub-command. This command uses a LLM to generate a set of keyword searches for OpenAlex. Then, for each keyword search it returns results that are sorted in ascending and descending citation counts, publication year, and it performs a random sample search. Each result is scored by vector similarity to the original query, and the top results are printed.

#+BEGIN_SRC sh
litdb lsearch "some query of interest"
#+END_SRC

some options include:

- -q :: integer count of number of keyword queries to generate
- -n :: integer count of results to retrieve for each keyword query
- -k :: integer count of how many results to display

** One-time additions of articles to litdb

You add an article by its DOI. There are optional arguments to also add references, citing and related articles.

#+BEGIN_SRC sh
litdb add doi --references --citing --related
#+END_SRC

To add an author, use their orcid. You can use ~litdb author-search firstname lastname~ to find an orcid for a person.

#+BEGIN_SRC sh
litdb add orcid
#+END_SRC

To add entries from a bibtex file, use the path to the file.

#+BEGIN_SRC sh
litdb add /path/to/bibtex.bib
#+END_SRC

You can provide more than one source and even mix them like this.

#+BEGIN_SRC sh
litdb add doi1 doi2 orcid
#+END_SRC

These are all one-time additions.

You can also add things like YouTube videos and podcasts. We use ML to extract the audio from these to text so they become searchable!

*** Adding references from text

You can paste text containing academic references and litdb will use an LLM to extract and add them automatically. This is useful when copying reference lists from PDFs, papers, or websites.

#+BEGIN_SRC sh
litdb fromtext "Kitchin, Examples of Effective Data Sharing in Scientific Publishing, ACS Catalysis, 2015, DOI: 10.1021/acscatal.5b00538"
#+END_SRC

You can also add multiple references at once:

#+BEGIN_SRC sh
litdb fromtext "Recent work by Smith et al. (Nature, 2020) showed...
Another study by Jones (Science, 2019) demonstrated..."
#+END_SRC

The command supports the same options as ~litdb add~ for adding related works:

#+BEGIN_SRC sh
litdb fromtext "TEXT" --references --related --citing
#+END_SRC

You can also specify which LLM model to use:

#+BEGIN_SRC sh
litdb fromtext "TEXT" --model "gpt-4o"
#+END_SRC

The LLM extracts structured reference information, then litdb uses CrossRef to find DOIs and match references. High-confidence matches are added automatically, while lower-confidence matches prompt for confirmation.

See [[./FROMTEXT_USAGE.md][detailed documentation]] for more information on supported citation formats and how the matching works.

** Adding filters

litdb provides several convenient ways to add queries to update your litdb in the future.

*** Follow an author

To get new papers by an author, you can follow them.

#+BEGIN_SRC sh
litdb follow orcid
#+END_SRC

*** Watch a query

#+BEGIN_SRC sh
litdb watch "filter to query"
#+END_SRC

*** Citations on a paper

#+BEGIN_SRC sh
litdb citing doi
#+END_SRC

*** Related papers

#+BEGIN_SRC sh
litdb related doi
#+END_SRC

*** A custom filter

A filter is used in OpenAlex to search for relevant articles. Here is an example of adding a filter for articles in the journal Digital Discovery. This doesn't add any entries directly, it simply stores the filter in the database. The main difference of this vs the watch command above is the explicit description.

#+BEGIN_SRC sh
litdb add-filter "primary_location.source.id:https://openalex.org/S4210202120" -d "Digital Discovery"
#+END_SRC

*** Managing and updating the filters

You can get a list of your filters like this.

#+BEGIN_SRC sh
litdb list-filters
#+END_SRC

You can update the filters like this.

#+BEGIN_SRC sh
litdb update-filters
#+END_SRC

This adds papers that have been created since the last time you ran the filter. You need an OpenAlex premium API key for this. This will update the last_updated field.

You can remove a filter like this:

#+BEGIN_SRC sh
litdb rm-filter "filter-string"
#+END_SRC

** Review your litdb

I find it helpful to review your litdb. To get a list of articles added in the last week, you can run this command.

#+BEGIN_SRC sh
litdb review -s "1 week ago"
#+END_SRC

This works best when you update your litdb regularly. You might want to redirect that into a file so you can review it in an editor of your choice.

** Generating newsletter summaries

You can generate a newsletter-style summary of recently added articles. This command uses an LLM to analyze articles, extract topics, aggregate them into themes, and create narrative summaries organized by topic.

#+BEGIN_SRC sh
litdb summary -s "1 week" -o newsletter.org
#+END_SRC

The default time period is "1 week", but you can specify different periods using natural language:

#+BEGIN_SRC sh
litdb summary -s "2 weeks"
litdb summary -s "1 month"
litdb summary -s "3 days"
#+END_SRC

You can specify which LLM model to use (defaults to the model in your ~litdb.toml~ config):

#+BEGIN_SRC sh
litdb summary -s "1 week" --model "gpt-4o"
litdb summary -s "2 weeks" --model "anthropic/claude-3-5-sonnet-20241022"
#+END_SRC

If you omit the ~-o~ option, the summary is written to a file with an auto-generated name based on the date range.

The output is an org-mode formatted newsletter with:
1. Main topic sections (5-10 automatically identified themes)
2. Subtopic subsections with narrative summaries
3. Numbered references within the narrative text
4. Bibliography of articles for each subtopic

This command works best with articles that have abstracts. It only processes entries classified as articles (journal articles, reviews, preprints, etc.), skipping other entry types like books or datasets.

The process:
1. Fetches articles added since the specified date
2. Extracts 3-5 topics from each article using an LLM
3. Aggregates all topics into 5-10 main themes with subtopics
4. Classifies each article into the most relevant topic/subtopic
5. Generates narrative summaries for each subtopic, referencing the articles
6. Formats everything as an org-mode document

** Searching litdb

There are several search options.

*** vector search

The main way litdb was designed to be searched is with by natural language queries. The way this works is your query is converted to a vector using SentenceTransformers, and then a vector search identifies entries in the database that are similar to your query.

#+BEGIN_SRC sh
litdb vsearch "natural language query"
#+END_SRC

The default number of entries returned is 3. You can change that with an optional argument

#+BEGIN_SRC sh
litdb vsearch "natural language query" -n 5
#+END_SRC

There is an iterative version of vsearch called isearch. This finds the closest entries, then downloads the citations, references and related entries for each one, and repeats the query until you tell it to stop, or it doesn't find any new results.

#+BEGIN_SRC sh
litdb isearch "some query"
#+END_SRC

*** full text search

There is a full text search (full on the text in litdb) available. The command looks like this.

#+BEGIN_SRC sh
litdb fulltext "query"
#+END_SRC

See https://sqlite.org/fts5.html for information on what the query might look like. The search is done with this SQL command:

#+BEGIN_SRC sql
select source, text from fulltext where text match ? order by rank
#+END_SRC

The default number of entries returned is 3. You can change that with an optional argument

#+BEGIN_SRC sh
litdb fulltext "natural language query" -n 5
#+END_SRC

*** hybrid search

Vector and full text search have complementary strengths and weaknesses. We combine them in the hybrid-search subcommand. This performs two searches on two different queries, and combines them with a unified score that is used to rank all the matches. This ensures you get some results that match the full search, and the vector search. It is worth trying if you aren't finding what you want by vector or text search alone.

#+BEGIN_SRC sh
litdb hybrid-search "vector query" "text query"
#+END_SRC

*** ollama GPT

You can use litdb as a RAG source for ollama. This looks up the three most related papers to your query, and uses them as context in a prompt to ollama (with the llama2 model). I find this quite slow (it can be minutes to generate a response on an old Intel Mac). I also find it makes up things like references, and that it is usually necessary to actually read the three papers. The three papers come from the same vector search described above.

#+BEGIN_SRC sh
litdb gpt "what is the state of the art in automated laboratories for soft materials"
#+END_SRC

*** Integration with litellm

litdb supports litellm so you can use almost any LLM provider you want: OpenAI, Anthropic, Gemini, whatever you have an API key for.

The free tier of the API includes 1,500 requests per day with Gemini 1.5 Flash.

It uses a different command than the ollama gpt command.

#+BEGIN_SRC sh
litdb chat "what is the state of the art in automated laboratories for soft materials"
#+END_SRC

There are some fancy things you can do with the prompt:

1. Avoid using RAG if --norag is in your prompt.
2. If you surround python objects with backticks, it will try expanding that to the documentation from Python.
3. A line that starts with < indicates a shell command to run and the output will be expanded into the prompt.
4. A prompt of !save will save the current chat to a file.
5. You can use this syntax to expand a file or url in the prompt for context:
#+BEGIN_EXAMPLE
[[file/url]]
#+END_EXAMPLE

Your prompt history is saved in your litdb, so you can go back to them if you want.

*** Web app for litdb

If you prefer a browser, you can now launch a streamlit app for litdb:

#+BEGIN_SRC sh
litdb app
#+END_SRC

This should launch the app in your browser and you can search litdb from it. The terminal application is more advanced in terms of prompt expansion.

*** search with audio

This command will record audio, transcribe that audio to text, and then do a vector search on that text. You will be prompted when the recording starts, and you press return to stop it. litdb will show you what it heard, and ask if you want to do a vector search on it.

#+BEGIN_SRC sh
litdb audio -p
#+END_SRC

I haven't found the transcription to be that good on technical scientific terms. This is a proof of concept capability.

Note that you need to install these libraries for this feature to work:

pyaudio, playsound, SpeechRecognition

These are not trivial to install, and pyaudio relies on external libraries like portaudio that may not be easy to install. These are currently commented out in pyproject.toml because of these difficulties.

*** search from a screenshot

You can copy a screenshot to the clipboard, and then use OCR to extract text from it, and do a vector search on that text.

#+BEGIN_SRC sh
litdb screenshot
#+END_SRC

If you can copy and paste text, you should do that instead. This is helpful to get text from images, or pdfs where the text is stored in an image, maybe from videos, or screen share from online meetings, etc.

Eventually, if images get integrated into litdb, this is also an entry point for image searches.

** Tagging entries

litdb supports tagging entries so you can group them. To tag a source with tag1 and tag2, use this syntax.

#+BEGIN_SRC sh
litdb add-tag source -t tag1 -t tag2
#+END_SRC

You can remove tags like this.

#+BEGIN_SRC sh
litdb rm-tag source -t tag1 -t tag2
#+END_SRC

You can delete a tag from the database.

#+BEGIN_SRC sh
litdb delete-tag tag1
#+END_SRC

To see all the tags do this.

#+BEGIN_SRC sh
litdb list-tags
#+END_SRC

To see entries with a tag:

#+BEGIN_SRC sh
litdb show-tag tag1
#+END_SRC

You can use this to export tagged entries into bibtex entries like this.

#+BEGIN_SRC sh
litdb show-tag workflow -f '{{ source }}' | litdb bibtex
#+END_SRC

** Exporting entries

You can use these commands to export bibtex entries or citation strings.

*** Get a bibtex entry

This command will try to generate a bibtex entry for entries in your litdb.

#+BEGIN_SRC sh
litdb bibtex doi1 doi2
#+END_SRC

The output can be redirected to a file.

You can also use a search like this and pipe the output to litdb bibtex.

#+BEGIN_SRC sh
litdb vsearch "machine learning in catalysis
" -f "{{ source }}" | litdb bibtex
#+END_SRC


*** Get a citation string

This command will output a citation for the sources. It is mostly a convenience function. There is not currently a way to customize the citation.

#+BEGIN_SRC sh
litdb citation doi1 doi2
#+END_SRC

You can also use a search like this and pipe the output to litdb bibtex.

#+BEGIN_SRC sh
litdb vsearch "machine learning in catalysis
" -f "{{ source }}" | litdb citations
#+END_SRC

** Find free pdfs

You can use litdb to find freely available PDFs via https://unpaywall.org/.

#+BEGIN_SRC sh
litdb unpaywall doi
#+END_SRC

These do not always work, and sometimes you get a version from arxiv or pubmed.

** Low-level interaction with litdb

litdb is just a sqlite database (although you need to use the libsql executable for vector search). There is a CLI way to run a sql command. For example, to find all entries with a null bibtex field and their types use a query like this.

#+BEGIN_SRC sh
litdb sql "select source, json_extract(extra, '$.type'), json_extract(extra, '$.bibtex') as bt from sources where bt is null"
#+END_SRC

You might also use this for very specific queries. For example, here I search the citation strings for my name.

#+BEGIN_SRC sh
litdb sql "select source, json_extract(extra, '$.citation') as citation from sources where citation like '%kitchin%'"
#+END_SRC

** Deep research

Deep research is one of the newer ideas in LLMs. It is a combination of web and other searches to find relevant information, coupled with an LLM to synthesize that information into a textual response. Litdb supports deep research with your database using [[https://docs.gptr.dev/][gpt-researcher]] with this command:

#+BEGIN_SRC sh
litdb research "some query"
#+END_SRC

litdb will generate some documents from your database from your query, and use this in a research task with gpt_researcher. This takes longer than a simple chat, but is often much richer in detail, and summarizes across documents.

The default setup is to use a local ollama model, with arxiv as a search tool. You will get better results most likely if you use a cloud LLM. To use these you can add this new section to litdb.toml:

#+BEGIN_SRC text
[gpt-researcher]

FAST_LLM = "google_genai:gemini-2.0-flash"
SMART_LLM = "google_genai:gemini-2.0-flash"
STRATEGIC_LLM = "google_genai:gemini-2.0-flash"
EMBEDDING = "google_genai:models/text-embedding-004"
#+END_SRC

See https://docs.gptr.dev/docs/gpt-researcher/llms/llms  for other configurations.

The default search engine in litdb for deep research is DuckDuckGo and arxiv. If you setup these API keys you can also use pubmed (https://www.ncbi.nlm.nih.gov/account/), google (https://developers.google.com/custom-search/v1/overview), and  [[https://app.tavily.com/home][Tavily]] for search.

#+BEGIN_SRC sh
export NCBI_API_KEY=
export GOOGLE_CX_KEY=
export TAVILY_API_KEY=
#+END_SRC

I have not tried the other retrievers at https://github.com/assafelovic/gpt-researcher/tree/master/gpt_researcher/retrievers.

You can do research on local documents with this command:

#+BEGIN_SRC sh
litdb research "some query" --doc-path=/path/to/documents
#+END_SRC

It is not currently possible to use litdb do research with and use a local directory full of documents at the same time.

The default is to print the report to stdout. To write to a file instead use:

#+BEGIN_SRC sh
litdb research "some query" -o report.md
#+END_SRC

You specify other output formats, anything pandoc can convert to. However, I found pdf a challenge for pypandoc, and use md2pdf for that instead. On Mac, if you see errors related to gobject, you probably have do run something like this to fix it.

#+BEGIN_SRC sh
brew list pango gobject-introspection
export DYLD_LIBRARY_PATH=$(brew --prefix)/lib:$DYLD_LIBRARY_PATH
#+END_SRC

** Extracting tables from PDFs

You can extract tables from PDF files using the ~extract~ command. This uses computer vision and machine learning to identify and extract tables.

#+BEGIN_SRC sh
litdb extract paper.pdf
#+END_SRC

This will extract all tables from the PDF. To extract specific tables (using 1-based indexing):

#+BEGIN_SRC sh
litdb extract paper.pdf -t 1 -t 3
#+END_SRC

This extracts only tables 1 and 3. You can specify an output format with the ~-f~ option:

#+BEGIN_SRC sh
litdb extract paper.pdf -t 1 -f markdown
#+END_SRC

The extracted tables are returned as pandas DataFrames, which can be formatted in various ways.

** Extracting structured data with schemas

You can extract structured information from documents (PDFs, web pages, etc.) using the ~schema~ command. This uses an LLM to parse the document and extract data according to a schema you define.

*** Using the schema DSL

The schema command supports a simple domain-specific language (DSL) for defining schemas:

#+BEGIN_SRC sh
litdb schema paper.pdf "title:str, authors:list, year:int, doi?:str"
#+END_SRC

Schema DSL syntax:
- ~fieldName:type~ - Required field with type
- ~fieldName?:type~ - Optional field (can be None)
- ~fieldName:type=defaultValue~ - Field with default value
- ~fieldName?:type=defaultValue~ - Optional field with default

Supported types: ~str~, ~int~, ~float~, ~bool~, ~list~, ~dict~

Examples:

#+BEGIN_SRC sh
# Extract basic paper metadata
litdb schema paper.pdf "title:str, authors:list, year:int"

# Extract with optional DOI and default value for journal
litdb schema paper.pdf "title:str, doi?:str, journal:str=Unknown"

# Extract experimental data
litdb schema data.pdf "temperature:float, pressure:float, yield:float, catalyst:str"
#+END_SRC

*** Using JSON schemas

You can also provide a schema as a JSON object:

#+BEGIN_SRC sh
litdb schema paper.pdf '{"title": ["str", None], "year": ["int", 2024]}'
#+END_SRC

The schema command first converts the document to markdown using Docling, then uses an LLM with structured output to extract the requested fields. The LLM model used is configured in the ~[llm]~ section of your ~litdb.toml~ file.

** Claude Desktop + MCP

#+BEGIN_SRC sh
litdb_mcp install /path/to/litdb.libsql
#+END_SRC

Then, restart Claude Desktop.

Now in the Claude Desktop app you can use prompts like "Search my litdb for references on alloy segregation" or "Search OpenAlex for articles on alloy segregation."


* Adding local files

The idea of using local files is that it is likely you have collected information in the form of files on your hard drive, and you want to be able to find information in those files.

It is possible to add any file that can be turned into text to litdb. That includes:

- pdf
- docx
- pptx
- html
- ipynb
- org / md
- bib
- url

This limits portability because you need a path if you want to be able to open that file.

The same vector, fulltext and gpt search commands are available for local file entries. These tend to be longer documents than the OpenAlex entries, and I am not sure how well the search works at the document level embeddings. Search at a chunk level is very precise; odds are you want paragraph level similarity to your query.

An early version of litdb stored each chunk. This is possible, but I used another table for it. You could munge the source to be something like f.pdf::chunk-1 so each one is unique, but that seems more complicated and you would need to do some experiments to see if it is warranted.

You can combine this with the OpenAlex entries in a single database.

You can walk a directory and add files from it with this command.

#+BEGIN_SRC sh
litdb index dir1
#+END_SRC

This directory is saved and you can update all the previously indexed directories like this.

#+BEGIN_SRC sh
litdb reindex
#+END_SRC

Some annoying things that may happen are duplicate content, e.g. because you have the same file in multiple formats like docx and pdf, or because you have literal copies of files in multiple places.

You should also be careful sharing a litdb that has indexed local files. It may have sensitive information that you don't want others to be able to find.

* Emacs integration

Of course there is some Emacs integration. I made a new link for litdb.

[[litdb:https://doi.org/10.1021/jp047349j]]

The links export as \cite{source}, and there is a function ~litdb-generate-bibtex~ to export bibtex entries for all links in the buffer. These entries are not certain to be valid, most likely from the keys (some DOIs are probably invalid keys).

You can easily insert a link like this:

M-x litdb

See [[./litdb.el]] for details. This is not a package on MELPA yet. You should just load the .el file in your config. You can also use ~litdb-fulltext~, ~litdb-vsearch~, and ~litdb-gpt~ from Emacs to interact with your litdb.

litdb.el is under active development, and will be an alternative UI to the terminal eventually. It is too early to tell if it will replace org-ref. It has potential, but that would be a very large undertaking.

* Database design

litdb uses a sqlite database with libsql. libsql is a sqlite fork with additional capabilities, most notably integrated vector search.

The main table in litdb is called sources.
- sources
  - source (url to source location)
  - text (the text for the source)
  - extra (json data)
  - embedding (float32 blob in bytes)
  - date_added string

This table has an embedding_idx index for vector search.

There is also a virtual table fulltext for fulltext search.

- fulltext
  - source
  - text

And a table called queries.
- queries
  - filter
  - description
  - last_updated

This database is automatically created when you use litdb.

* Limitations

The text that is stored for each entry comes from OpenAlex and is typically limited to the title and abstract. For the text in each entry The first line is typically a citation including the title, and the rest is the abstract if there is one. I feel like I see more and more entries with no abstract. This will certainly limit the quality of search, and could bias results towards entries with more text in them.

The quality of the vector search depends on several things. First, litdb stores a document level embedding vector that is computed by averaging the embedding vectors of overlapping chunks. We use Sentence Transformers to compute these. There are many choices to make on the model, and these have not been tested exhaustively. So far 'all-MiniLM-L6-v2' works well enough. There are other models you could consider like getting embeddings from ollama, but at the moment litdb can only use SentenceTransformers.

I guess that document level embeddings are less effective on longer documents. The title+abstract from OpenAlex is pretty short, and so far there isn't evidence this is a problem.

Second, we rely on defaults in libsql for the vector search, notably finding the top k nearest vectors based on cosine similarity. There are other distance metrics you could use like L2, but we have not considered these.

The query is based on vector similarity between your query and the texts. So, you should write the query so it looks like what you want to find, rather than as a question. It is less clear how you should structure your query if you are using the GPT capability. It is more natural to ask a question, or give instructions. The RAG is still done by similarity though.

Finally, the search can only find things that are in your database. If you haven't added it there, you won't find it. That definitely means you will miss some papers. I try to use a mesh of approaches to cover the most likely papers. This includes:

1. Follow authors
2. add references, related, and citing papers to the most relevant papers.
3. Use text search filters
4. Add papers I find from X, bluesky, LinkedIn, etc. (and their references, related, etc)
5. If read a paper in litdb that is good, add its references, related, etc.

It is an iterative process, and you have to make a judgment call about when to stop it. You can always come back later. There might even be newer papers to find.

** Local file limitations

Similar limitations exist for local files. There are additionally the following known limitations:

1. The quality of document to text influences the ultimate embedding. This varies by type of document, and the library used to convert it.
2. Local files tend to be longer documents and this can lead to hundreds of text chunks per document. These chunk embeddings are averaged into one embedding. It is not obvious this is as effective as vector search on each chunk, but it is more memory efficient.

For PDF to text we use [[https://pymupdf.readthedocs.io/en/latest/pymupdf4llm/][pymupd4fll]] which works for this proof of concept. There is a Pro version of that package which supports more file formats. It is not obvious what it would cost to use that. I used [[https://ds4sd.github.io/docling/][docling]] in an early prototype. It also worked pretty well, but it was a little slower I think, and would occasionally segfault so I stopped using it.  Spacy is integrating PDF to structured data using docling (https://explosion.ai/blog/pdfs-nlp-structured-data). There is plenty of room for improvement in this dimension, with trade offs in performance and accuracy.

There is a new package from Microsoft to convert Office files to Markdown (https://github.com/microsoft/markitdown) that they specifically mention using in the context of LLMs.

The embedding model we use is trained on text. It is probably not as good at finding code, and the gpt we use is also probably not good at generating code. I guess you would need another table in the database for code, and a different model for embedding and generation. This only matters if you index jupyter notebooks (and later if other code files are supported).

** sqlite + sqlite-vec vs libsql

Vector search is the core requirement for litdb. There are many ways to achieve this. I only considered local solutions so the options are:

- sqlite + vectorlite (https://github.com/1yefuwang1/vectorlite)
- sqlite + sqlite-vec (https://github.com/asg017/sqlite-vec)
- libsql https://github.com/tursodatabase/libsql

vectorlite aims to be faster than sqlite-vec, but it relies on hnsw for vector search, and I was uncomfortable figuring out how to set the size of the db for this application.

sqlite-vec is nice, and early versions of litdb used it and its precursor. This approach requires a virtual table for the embeddings. This is installed as an extension, and is still considered in early stages of adoption.

libsql is a fork of sqlite with integrated vector search, and potential for using it as a cloud database. It is supported by a company, with freemium cloud services. In libsql you store the vectors in a regular table, and search on an embedding index. The code is on GitHUB, and can also be used locally.

* Roadmap

These are ideas for future expansion.

** PDFs and notes

I am not sure what the best way to do this is. The records in litdb are stored by the source, often a url, or path. The PDFs would be stored outside the database, and we would need some way to link them. The keys aren't suitable for naming, but maybe a hash of the keys would be suitable. This would add a fuller opportunity to search larger, local documents too. In org-ref, I only had one pdf per entry. I guess here I would have a new table, so you could have multiple documents linked to an entry, although it won't be easy to tell what they are from the hash-based filenames.

Notes on the other hand, might be small enough that they could be stored in the database. Then they would be easily searchable. They could also be stored externally to make them easy to edit. I haven't found the notes feature in org-ref that helpful, and usually I take notes in various places. What I should do is add a search to find the litdb links in your org-files. This is already a feature of org-db.

** Jupyter lab integration

An alternative to the CLI and Emacs would be to run this in Jupyter Lab with magic commands and rich output.

** graph visualization

It might be helpful to have a graph representation of a paper that shows nodes of citing, references, and related papers, with edge length related to a similarity score, and node size related to number of citations.

ResearchRabbit and Litmaps do this pretty well.

** ollama and agents

There might be a way to get better results using agents and / or tools. For example, you might have a tool that can lookup new articles on OpenAlex, or augment with google search somehow. Or there might be some iterative prompt building tool that refines the search for related articles based on output results.

Here are some references for when I get back to this.

- https://github.com/ollama/ollama-python
- https://github.com/MikeyBeez/Ollama_Agents
- https://github.com/premthomas/Ollama-and-Agents
- https://medium.com/@lifanov.a.v/integrating-langgraph-with-ollama-for-advanced-llm-applications-d6c10262dafa
- https://medium.com/@abhilasha.sinha/building-a-crew-of-agents-with-open-source-llm-using-ollama-to-analyze-fund-documents-as-multi-page-756d8fd9fbf0
- https://blog.paperspace.com/building-local-ai-agents-a-guide-to-langgraph-ai-agents-and-ollama/

I don't use llamaindex (maybe I should see what it does), but it has this section on agents https://docs.llamaindex.ai/en/stable/understanding/agent/

** web app / fast-api

It might be nice to have a flask app with an API. This would facilitate interaction with Emacs.

** async operations

Almost everything is done synchronously and it blocks the program. At least some things could be done asynchronously I think, and that might speed things up (especially for local files), or at least let you do other things while it happens.

The only thing to be careful about is not exceeding rate limits to OpenAlex. This is handled in the synchronous code.

** application specific encoders

I use a generic embedding model, and there are others that are better suited for specific tasks. For example:

- MatBERT [[cite:&trewartha-2022-quant-advan]]
- Scibert [[cite:&beltagy-2019-sciber]]
- Matscibert [[cite:&gupta-2022-matsc]]
- Specter cite:&cohan-2020-spect https://www.sbert.net/docs/sentence_transformer/pretrained_models.html#scientific-similarity-models
- PaECTER [[cite:&ghosh-2024-paect]] for patents

These might have a variety of uses with litdb that range from extracting data, named entity recognition, specific searches on materials, etc.

It is not essential to use SentenceTransformers for embedding, they are just easy to use. An alternative is something like ollama embeddings (https://ollama.com/blog/embedding-models) or llama.cpp https://github.com/abetlen/llama-cpp-python?tab=readme-ov-file#embeddings. The main reason to use on of these would be performance, and maybe better integration with a chat llm.

It is not that easy to just switch models; you would need to either add new columns and compute embeddings for everything, or update all the embeddings for a new model. The SPECTER embedding is much bigger than the all-MiniLM-L6-v2 embedding.

#+BEGIN_SRC jupyter-python :restart
from sentence_transformers import SentenceTransformer

m = SentenceTransformer('allenai-specter')
print(m.encode(['test']).shape)
#+END_SRC

#+RESULTS:
: (1, 768)


** merge databases

I have setup litdb to be project based. There may come a time when it is desirable to merge some set of databases. It might not be necessary, I think you can attach databases in sqlite (https://www.sqlitetutorial.net/sqlite-attach-database/) to achieve basically the same effect. litdb doesn't store version info at the moment, so it could be tricky t