EvoRM: An Evolvable Neuro-Symbolic Reasoning Framework for LLM-Assisted Entity Matching

Paper: EvoRM: An Evolvable Neuro-Symbolic Reasoning Framework for LLM-Assisted Entity Matching
Venue: IEEE Transactions on Knowledge and Data Engineering (TKDE), Initial Submission
Repository: Official paper companion — code, datasets, and experimental results

Abstract

Entity Matching (EM) is a fundamental task in data integration that identifies whether two data instances refer to the same real-world entity. Recent advances in Large Language Models (LLMs) have shown promise for EM, but they suffer from two critical limitations: (1) high computational cost from repeatedly invoking LLMs for each entity pair, and (2) lack of systematic knowledge accumulation across matching decisions. We propose EvoRM, an evolvable neuro-symbolic reasoning framework that integrates symbolic rule learning with LLM-based reasoning. EvoRM consists of four tightly integrated components: RuleEncoding extracts First-Order Logic (FOL) rules from LLM reasoning traces; HypergraphStorage organizes rules in a weighted hypergraph structure enabling efficient retrieval; TwoStageInferenceController employs a lightweight MLP gate to route entity pairs between symbolic filtering (Stage-1) and LLM refinement (Stage-2); and RuleMaintenance dynamically manages rule confidence, freshness, and pruning. EvoRM operates as a plug-and-play plugin that can enhance any LLM-based EM method without modifying its internal architecture. Extensive experiments on 17 datasets across four tasks — Entity Resolution, Entity Alignment, Entity Linking, and Schema Matching — demonstrate that EvoRM achieves +6.8% to +12.3% F1 improvement over baseline methods while reducing LLM API calls by up to 40%.

Key Concepts

The Four Components

Component	Paper Section	Function
RuleEncoding	III-C	Extracts FOL rules from LLM rationales using template-based parsing (SameValue, DifferValue, ShareNeighbor, DifferNeighbor) + LLM-based semantic parsing (SemanticEquiv, SemanticConflict)
HypergraphStorage	III-D	Weighted hypergraph H = (V, E) with entity/attribute/value nodes (V = V_E ∪ V_A ∪ V_V) and hyperedges encoding rules; invert-indexed for O(1) retrieval
TwoStageInferenceController	III-E	Stage-1: symbolic FOL rule verification → direct MATCH/NON-MATCH routing. MLP Gate g_ϕ routes Survival pairs to Stage-2: LLM refinement with evidence subgraph
RuleMaintenance	III-F	Dynamic confidence (Conf(R)) and freshness (Fresh(R)) tracking; rule pruning (θ_f), conclusion flipping (θ_c), merging (η_m), and capacity-based eviction

Plug-and-Play Architecture

EvoRM is designed as a drop-in plugin for any LLM-based entity matching method:

# Without EvoRM (original baseline)
response = client.chat.completions.create(
    model="gpt-3.5-turbo-1106",
    messages=[{"role": "user", "content": prompt}],
)

# With EvoRM (plug-and-play)
from evorm_plugin import EvoRMPlugin
evorm = EvoRMPlugin(client=client)
result = evorm.stage2_simple(entity_id_a, entity_id_b, entity_a_ctx, entity_b_ctx, [], prompt)

Repository Structure

EvoRM/
├── README.md                          # This file
├── evorm_plugin.py                    # Core EvoRM framework (~1700 lines)
│   ├── RuleEncoding                   #   FOL rule extraction + parsing
│   ├── HypergraphStorage              #   Weighted hypergraph + inverted index
│   ├── TwoStageInferenceController    #   Stage-1 symbolic + Stage-2 LLM
│   └── RuleMaintenance                #   Confidence/freshness/pruning/merging
├── evorm_mlight.py                    # Mlight (rationale) + Mheavy (decision) pipeline
├── evorm_entity_embedding.py          # Entity embeddings (d_emb=1024, feature hashing)
├── evorm_mlp_gate.py                  # MLP Gate g_ϕ (self-supervised training)
├── evorm_config.py                    # Centralized hyperparameter configuration
│
├── baselines/                         # Baseline implementations
│   ├── evorm_wrappers/                # EvoRM wrappers for baselines
│   │   ├── chat_ea.py                # ChatEA + EvoRM (Entity Alignment)
│   │   ├── matchgpt.py               # MatchGPT/Anymatch + EvoRM (Entity Resolution)
│   │   ├── lela_el.py                # LELA + EvoRM (Entity Linking + Schema Matching)
│   │   ├── zero_cot.py               # Zero-shot CoT + EvoRM
│   │   ├── self_consistency.py       # Self-Consistency + EvoRM
│   │   └── cohard.py                 # CoHard + EvoRM
│   └── original/                     # Original baseline implementations
│       ├── ChatEA/                   # ChatEA (entity alignment)
│       └── LELA/                     # LELA (entity linking)
│
├── experiments/                       # Experiment scripts
│   ├── run_optimized_experiments.py  # Full 102-experiment suite
│   ├── run_ablation.py               # Ablation study (6 modes)
│   ├── run_baselines_v2.py           # Baseline comparison
│   └── run_complete_v2.py            # Complete experiment pipeline
│
├── data/                              # Datasets
│   ├── er/                           # Entity Resolution (9 datasets)
│   │   ├── Structured/               # Amazon-Google, Beer, DBLP-ACM, DBLP-Scholar,
│   │   ├── Dirty/                    #   Fodors-Zagats, Walmart-Amazon, iTunes-Amazon
│   │   └── Textual/                  # Abt-Buy, Company
│   ├── ea/                           # Entity Alignment
│   │   └── dbp15k/                   # DBP15K (zh_en, fr_en, ja_en)
│   ├── el/                           # Entity Linking
│   │   ├── AIDA-CoNLL_real.json      # AIDA-CoNLL
│   │   ├── WNED-CWEB_real.json       # WNED-CWEB
│   │   └── el_benchmark/             # ZESHEL domains
│   ├── sm/                           # Schema Matching
│   │   ├── T2Dv2_schema.json
│   │   ├── DBP15K_schema.json
│   │   └── ER_schema.json
│   ├── icews_wiki/                   # ICEWS-WIKI (HHEA, 3,540 ref pairs)
│   └── icews_yago/                   # ICEWS-YAGO
│
├── results/                           # Experiment results
│   ├── optimized_results_20260804_004335.json  # 102 experiments (latest)
│   ├── complete_results_v2_merged.json         # 161 experiments
│   ├── ablation_6modes_v3_20260801_053823.json # Ablation study
│   └── RESULTS_TABLES.md             # Formatted results tables
│
├── Area2/                            # AdaCoAgentEA integration
│   ├── LLM1_label_selector.py        # With EvoRM integration
│   ├── LLM1_label_selector_baseline.py # Baseline (no EvoRM)
│   └── LLM1_label_selector_evorm.py  # EvoRM-only variant
│
└── AdaCoAgentEA_20260804.tar.gz      # Full packaged archive (1.4 GB)

Quick Start

Prerequisites

pip install openai tqdm

Running a Single Experiment

from baselines.evorm_wrappers.matchgpt import run_matchgpt

# Entity Resolution — MatchGPT with EvoRM
results = run_matchgpt(
    data_path="data/er_benchmark/beer.json",
    backend="default",        # default, mixtral, solar, beluga2
    use_evorm=True,
    max_pairs=50,
)
print(f"F1: {results['F1']:.4f}, Precision: {results['Precision']:.4f}, Recall: {results['Recall']:.4f}")

Running the Full Experiment Suite

cd /root/autodl-tmp/AdaCoAgentEA
python run_optimized_experiments.py

This runs 102 experiments covering:

ER: MatchGPT (4 backends) + AnyMatch, × 8 datasets, with/without EvoRM
EL: LELA on AIDA-CoNLL, WNED-CWEB, with/without EvoRM
SM: llm_dp, rematch, matchmaker, with/without EvoRM

Results Summary

Entity Resolution (ER) — Synthetic Data (max_pairs=50)

Method	ABT	AMGO	BEER	DBAC	DBGO	FOZA	ITAM	WAAM	Avg
MatchGPT[default]	1.000	1.000	1.000	1.000	1.000	1.000	1.000	1.000	1.000
MatchGPT[default] + EvoRM	1.000	1.000	1.000	1.000	1.000	1.000	1.000	1.000	1.000
MatchGPT[solar]	0.973	0.982	1.000	1.000	0.982	1.000	0.982	0.985	0.988
MatchGPT[solar] + EvoRM	0.982	1.000	1.000	1.000	1.000	1.000	0.982	1.000	0.996
AnyMatch	1.000	1.000	1.000	1.000	1.000	1.000	1.000	1.000	1.000
AnyMatch + EvoRM	1.000	1.000	1.000	1.000	1.000	1.000	1.000	1.000	1.000

Entity Linking (EL) — Synthetic Data

Method	AIDA-CoNLL	WNED-CWEB
LELA	0.920	0.980
LELA + EvoRM	0.920	0.980

Schema Matching (SM) — Synthetic Data

Method	T2Dv2	MIMIC	SYNTHEA
llm_dp	1.000	1.000	1.000
llm_dp + EvoRM	1.000	1.000	1.000
rematch	0.979	1.000	1.000
rematch + EvoRM	0.979	1.000	1.000

Entity Alignment (EA) — ICEWS-WIKI (Real Data)

Method	Hits@1	Precision	Recall	F1
AdaCoAgentEA (Baseline)	0.9003	0.9235	0.9003	0.9117
AdaCoAgentEA + EvoRM (Full)	0.9508	0.99	0.9508	0.97

Ablation Study (ICEWS-WIKI)

Configuration	Hits@1	Precision	Recall	F1
Full System (EvoRM)	0.9267	0.9929	0.9267	0.9587
w/o Stage-1	0.9500	0.9965	0.9500	0.9727
w/o Maintenance	0.9433	0.9930	0.9433	0.9675

Cold-Start Scaling (RQ5)

N_warmup	Hits@1	Tokens/Pair
100	0.970	858.1
200	0.935	881.1
500	0.936	884.9
1000	0.938	877.9

Hyperparameter Configuration

All hyperparameters are centralized in evorm_config.py:

Parameter	Default	Paper Section	Description
`θ_gate`	0.5	III-E	MLP Gate decision threshold
`θ_f`	0.1	III-F	Freshness pruning threshold
`θ_c`	0.3	III-F	Confidence threshold for conclusion flipping
`η_m`	0.8	III-F	Jaccard threshold for rule merging
`η_h`	0.6	III-D	Jaccard threshold for hyperedge creation
`α`	0.6	III-D	Symbolic weight in hyperedge weight formula
`β`	0.4	III-D	Neural weight in hyperedge weight formula
`λ`	0.01	III-F	Decay rate for freshness
`d_emb`	1024	III-D	Entity embedding dimension
`	R	_max`	500
`γ`	0.2	III-F	Eviction ratio
`K`	5	III-E	Top-K evidence rules for Stage-2
`n_warmup`	20	III-E	MLP Gate warm-up rounds

Known Limitations

Paper vs Code Differences

Item	Paper Specification	Current Implementation	Status
MLP Gate (g_ϕ)	Self-supervised training with entity embeddings + rule features + topological features	Implemented in `evorm_mlp_gate.py` but disabled by default (requires CUDA); training requires ≥20 warm-up rounds	⚠️ Limited
Mlight/Mheavy Separation	Mlight = Qwen-2.5-3B, Mheavy = Llama-3.1-8B	Both use gpt-3.5-turbo-1106 via unified API	⚠️ Simplified
Entity Embeddings	Pre-computed d_emb=1024 embeddings	Feature hashing with 8192-dim random projection	⚠️ Simplified
LLM API	Multiple models (Llama, Qwen, Mixtral, SOLAR, Beluga2, GPT-3)	Single API: gpt-3.5-turbo-1106	⚠️ Limited
ER/EL/SM Datasets	17 real benchmark datasets	Synthetic data for ER/EL/SM (max_pairs=50); DBP15K + ICEWS-WIKI for EA	⚠️ Limited
Stage-1 Hit Rate	Expected to route 30-40% of pairs	~0% (synthetic data, no cross-dataset rule persistence)	⚠️ Limited
Paper Appendix I2/II2	T_enc, T_dec templates; MLP Gate architecture; hyperparameter tuning	Appendix missing from submission PDF	❌ Unavailable

Important Notes

Synthetic Data: ER/EL/SM experiments use synthetically generated data (max_pairs=50). Real dataset experiments require downloading and preprocessing the full benchmark datasets.
Stage-1 Not Firing: The persistence directory is reset between datasets, preventing cross-dataset rule transfer. For production use, configure a shared persistence_dir.
API Key: The hardcoded API key is for the LLM proxy service used during development. Replace with your own OpenAI/LLM API key.
EL Multi-Class: Entity Linking is a multi-class problem (C1/C2/.../NIL). EvoRM Stage-1 binary routing is skipped for EL tasks.

Reproducing Paper Results

Step 1: Prepare Real Datasets

# Download ER datasets (DeepMatcher format)
# Download DBP15K for EA
# Download ZESHEL for EL
# Download T2Dv2, MIMIC, SYNTHEA for SM
python prepare_new_datasets.py --all

Step 2: Run Experiments

# Full experiment suite (102 experiments)
python run_optimized_experiments.py

# Ablation study
python experiments/run_ablation.py --modes all

# Single task
python -c "
from baselines.evorm_wrappers.matchgpt import run_matchgpt
results = run_matchgpt('data/er_benchmark/beer.json', backend='default', use_evorm=True, max_pairs=100)
print(results)
"

Step 3: Generate Results Tables

python experiments/generate_tables.py --results results/optimized_results_20260804_004335.json

EvoRM Plugin API Reference

`EvoRMPlugin` (from `evorm_plugin.py`)

class EvoRMPlugin:
    def __init__(self, client, ablation_mode=None, persistence_dir='/tmp/evorm',
                 config=None, enable_mlp_gate=False):
        """Initialize EvoRM plugin.
        Args:
            client: OpenAI-compatible client
            ablation_mode: None, 'no_stage1', 'no_maintenance', 'no_mlp_gate'
            persistence_dir: Directory for rule persistence
            config: EvoRMConfig override
            enable_mlp_gate: Enable MLP Gate (requires CUDA + warm-up)
        """

    def stage1(self, entity_id_a, entity_id_b, entity_a_ctx, entity_b_ctx):
        """Stage-1: Symbolic rule-based filtering.
        Returns: (decision, triggered_rules, candidate_edges)
        - decision=0: NON-MATCH, decision=1: MATCH, decision=None: SURVIVAL
        """

    def stage2_simple(self, entity_id_a, entity_id_b, entity_a_ctx, entity_b_ctx,
                      triggered_rules, prompt):
        """Stage-2: LLM refinement with evidence subgraph.
        Returns: dict with 'raw_response', 'decision', 'rationale', 'rule_feedback'
        """

    def record_trajectory(self, entity_id_a, entity_id_b, entity_a_ctx, entity_b_ctx,
                          decision, rationale, triggered_rules, rule_feedback):
        """Record a decision trajectory for rule learning."""

    def get_stats(self):
        """Get runtime statistics: stage1_hits, llm_calls_saved, num_rules, etc."""

    def save(self, path):
        """Persist plugin state to disk."""

    @classmethod
    def load(cls, path, client):
        """Load plugin state from disk."""

Citation

If you use this code or datasets in your research, please cite:

@article{evorm2024tkde,
  title={EvoRM: An Evolvable Neuro-Symbolic Reasoning Framework for LLM-Assisted Entity Matching},
  author={[Authors]},
  journal={IEEE Transactions on Knowledge and Data Engineering},
  year={2024},
  note={Initial Submission}
}

The EvoRM framework is integrated into AdaCoAgentEA (ICDE 2025):

@inproceedings{adacoagentea2025,
  title={AdaCoAgentEA: Adaptive Collaboration of Multiple Agents for Entity Alignment},
  author={[Authors]},
  booktitle={Proceedings of the IEEE International Conference on Data Engineering (ICDE)},
  year={2025}
}

License

This project is licensed under the MIT License.

Contact

For questions about the paper or code, please open an issue on this repository or contact the authors.

This repository accompanies the TKDE submission "EvoRM: An Evolvable Neuro-Symbolic Reasoning Framework for LLM-Assisted Entity Matching". The code implements the neuro-symbolic reasoning framework described in the paper and serves as a reference for reproducing the experimental results.

Downloads last month: -; Downloads are not tracked for this model. How to track

Inference Providers NEW

This model isn't deployed by any Inference Provider. 🙋 Ask for provider support