SCTA paragraph TF-IDF (2026.10)
Word n-gram TF-IDF vectors of every paragraph of the Scholastic Commentaries and Texts Archive, built from scta/scta-corpus 2026.10 with scta-scikit. 1.2 GB.
It finds paragraphs that share phrasing with a text: quotations, borrowed passages, parallel wording. (For paragraphs about the same thing in other words, see the Doc2Vec models in scta/scta-doc2vec.)
- Features: word 2-grams, 3-grams and 3-of-4 skip-grams (v1), of the normalized text (lower case; u for v, i for j and y, e for ae and oe; letters only). The skip-grams let a phrase match with one word changed, added or left out.
- Hashed, not kept in a vocabulary: sklearn HashingVectorizer (MurmurHash3, alternate_sign=False) into 2,147,483,647 buckets. Buckets found in fewer than 2 paragraphs are dropped (they can't connect two paragraphs).
- Weights: raw tf x smooth idf, rows l2-normalized; a score is the cosine of two vectors.
- 672,989 paragraphs x 20,123,126 features, 120,876,251 nonzeros.
Files (models/para_tfidf/): matrix.npz (CSR, float32), buckets.npy (the bucket of each column),
idf.npy, ids.json and toplevels.json (each row's paragraph and text), info.json. No pickles.
Use
import scta_scikit as sc
sc.download(para_tfidf=True) # with the corpus it was built from
scta match "in principio erat verbum et verbum erat apud deum"
scta match --id <paragraph id>
Versions
A release is tagged with the corpus version it was built from (2026.10;
2026.10-2 for a rebuild of the same corpus); manifest.json names that corpus.
License
Derived from the SCTA corpus and published under the same license, Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International (CC BY-NC-SA 4.0), https://creativecommons.org/licenses/by-nc-sa/4.0/.
Built 2026-10-24T13:39:18Z with scta-scikit 055e6d1; see manifest.json.