unnicr/master-data-twin-finder

Master Data Twin Finder

Finds duplicate product and vendor records that are written differently, the kind that pile up in an ERP when different people type the same item or supplier their own way.

Record A	Record B	Verdict
PARLE-G BISCUIT 100GM	Parle G Bisc 100 g	✅ duplicate
Everest Haldi Powder 100g	EVEREST TURMERIC PWDR 100GM (MRP 45)	✅ duplicate
Sugar 1kg	Cheeni 1 Kilo	✅ duplicate
M/S SHARMA TRADERS PRIVATE LIMITED	Sharma Traders (P) Ltd.	✅ duplicate
Parle-G 100g	Parle-G 250g	❌ different size
Amul Taaza 1L x 6	Amul Taaza 1000ml	❌ different pack
Good Day Cashew 200g	Good Day Butter 200g	❌ different variant
Sharma Traders	Verma Traders	❌ different vendor

Try it: the web app runs entirely in your browser, so your data is never uploaded.

What makes it different

Plain fuzzy matching joins things that look alike. That merges 100 g with 250 g and Cashew with Butter, and it misses Haldi = Turmeric. This model is built for Indian retail and distribution master data:

• Size and pack aware. It reads 100gm, 0.1 kg, 1 Ltr, 1000 ML, x 6, Pack of 12 and 6 pcs, and converts units, so 1 kg = 1000 gm. Different sizes or packs are never merged.
• Abbreviation and typo tolerant. Bsct = Biscuit, Pwdr = Powder, Sunflwr = Sunflower, Cbues = Cubes.
• Hindi–English food words. haldi/turmeric, jeera/cumin, cheeni/sugar, namak/salt, chawal/rice, aata/atta, mirchi/chilli. These match even when misspelt (jera).
• Vendor suffix aware. It ignores M/s, Messrs, Pvt Ltd, (P) Ltd, Private Limited and LLP, and it understands Sri/Shri/Shree, Bros, Ent., Corpn, & Co and Genl Stores.
• Variant aware. A generic extra word (Cookies, Toned Milk, Detergent) is ignored. A distinctive extra word on both sides (Cashew vs Butter) means a different product.
• Safe grouping. Duplicates are grouped with average linkage plus a size guard, so a vague name like Parle G cannot chain 100 g and 250 g into one group. Groups where some names state no size are flagged check_size.
• Suggests which record to keep. It picks the spelling most of the group agrees with, so typo'd names are merged into the clean one.

How it works

name ──► normalize ──► 19 pair signals ──► gradient-boosted trees ──► probability ──► size-safe grouping

1. Normalize: lower-case, strip MRP/price, parse size into grams/ml, parse pack count, remove legal suffixes and filler words (pouch, jar, new), map synonyms, join split words (super market → supermarket).
2. Pair signals: character-trigram similarity (with and without spaces), soft word coverage from each side, unmatched distinctive words, first-word (brand) match, token-sort edit similarity, size equal/conflict/missing, pack conflict, number conflict.
3. Model: scikit-learn GradientBoostingClassifier, 250 trees of depth 4, exported to model.json.
4. Grouping: trigram blocking picks candidate pairs, then average-linkage merge above the threshold, never across different explicit sizes or packs.

twin_finder.py (Python) and space/twin.js (browser) are the same algorithm. test_parity.py checks that they give identical scores and groups.

Results

The test set uses products and vendors never seen during training.

Test	Result
Pair accuracy (5,100 unseen pairs)	99.3% (precision 99.5%, recall 98.9% for duplicates)
Different-size traps	100%
Different-pack traps	100%
Different-variant traps	96.7%
Similar vendor names (Sharma/Verma, Gupta/Goyal)	100%
Hand-written real-world cases	37 / 40
Grouping a messy 125-row product list	93.9% pair precision, 96.4% recall
Grouping a messy 73-row vendor list	100% precision, 100% recall

Full log: train_log.txt.

Known limits. When one name has an extra distinctive word and the other doesn't (Dairy Milk 50g vs Dairy Milk Silk 50g), the model can't know whether that word is a variant or just omitted text, so review these. Names without any size can't be placed with certainty and are flagged check_size. The model is trained on realistic synthetic data. Retraining with a few hundred labelled pairs from your own ERP will improve it further.

Use it

In the browser: open the web app, click Upload CSV, pick the name column and click Find twins, then Download result CSV.

From the command line (plain Python 3.8+, nothing to install):

python dedupe_csv.py m_product.csv                 # auto-detects the "Name" column
python dedupe_csv.py c_bpartner.csv --threshold 0.6 --out vendor_duplicates.csv

The report keeps your original columns (IDs, codes) and adds dup_group, keep, confidence and check_size.

Exporting from ADempiere / iDempiere:

SELECT M_Product_ID, Value, Name FROM M_Product
WHERE IsActive = 'Y' AND AD_Client_ID = <your client>;

SELECT C_BPartner_ID, Value, Name, TaxID FROM C_BPartner
WHERE IsActive = 'Y' AND IsVendor = 'Y' AND AD_Client_ID = <your client>;

In Python:

from twin_finder import TwinModel, find_duplicates

model = TwinModel.load("model.json")
print(model.score("PARLE-G BISCUIT 100GM", "Parle G Bisc 100 g"))   # ≈ 0.99

names = ["Tata Salt 1kg", "TATA NAMAK 1000 GM", "Tata Salt Lite 1kg", "Catch Salt 1kg"]
for g in find_duplicates(names, model):
    print([names[i] for i in g["members"]], "keep:", names[g["master"]])

Retrain

pip install -r requirements.txt
python generate_data.py      # creates data/*.csv
python train.py              # trains, evaluates, writes model.json
python test_parity.py        # checks the browser version matches (needs Node.js)

To teach it your own vocabulary, add words to SYNONYMS, GENERIC or STOPWORDS in twin_finder.py, or add your own labelled pairs to data/pairs_train.csv (name_a,name_b,label,kind), then retrain.

Files

File	Purpose
model.json	Trained model plus vocabulary, used by both Python and the browser
model.joblib	The same model as a scikit-learn object
twin_finder.py	Normalizer, features, scoring, duplicate grouping
dedupe_csv.py	Command-line tool for CSV exports
space/twin.js	Browser port (used by the web app)
generate_data.py, train.py, test_parity.py	Data generation, training and evaluation, parity test
data/	Training/test pairs and sample messy lists