metadata

license: apache-2.0

Use ColBERT as a search engine for the ACL Anthology. (Parse any bibtex, and store in a MySQL service)

Setup

Setup ColBERT

git clone https://huggingface.co/davidheineman/colbert-acl

# install dependencies
# torch==1.13.1 required (conda install -y -n [env] python=3.10)
pip install -r requirements.txt
brew install mysql

(Optional) Parse & Index the Anthology

Feel free to skip, since the parsed/indexed anthology is contained in this repo.

# get up-to-date abstracts in bibtex
curl -O https://aclanthology.org/anthology+abstracts.bib.gz
gunzip anthology+abstracts.bib.gz
mv anthology+abstracts.bib anthology.bib

# parse .bib -> .json
python parse.py

# index with ColBERT 
# (note sometimes there is a silent failure if the CPP extensions do not exist)
python index.py

Search with ColBERT

# start flask server
python server.py

# or start a production API endpoint
gunicorn -w 4 -b 0.0.0.0:8893 server:app

Then, to test, visit:

http://localhost:8893/api/search?query=Information retrevial with BERT

or for an interface:

http://localhost:8893

Deploy as a Docker App

docker-compose build --no-cache
docker-compose up --build

Example notebooks

To see an example of search, visit: colab.research.google.com/drive/1-b90_8YSAK17KQ6C7nqKRYbCWEXQ9FGs