Something’s Fishy in the Data Lake: A Critical Re-evaluation of Table Union Search Benchmarks (2025)

Abstract: Recent table representation learning and data discovery methods tackle table union search (TUS) within data lakes, which involves identifying tables that can be unioned with a given query table to enrich its content. These methods are commonly evaluated using benchmarks that aim to assess semantic understanding in real-world TUS tasks. However, our analysis of prominent TUS benchmarks reveals several limitations that allow simple baselines to perform surprisingly well, often outperforming more sophisticated approaches. This suggests that current benchmark scores are heavily influenced by dataset-specific characteristics and fail to effectively isolate the gains from semantic understanding. To address this, we propose essential criteria for future benchmarks to enable a more realistic and reliable evaluation of progress in semantic table union search.

## introduction measurement enables scientific progress. in computer science and machine learning, this requires the creation of efficient benchmarks that provide a stable foundation for evaluation, ensuring that observed performance scores reflect genuine capabilities for real-world tasks. table union search (tus) aims to retrieve tables c from a corpus that are semantically unionable with a query table q, meaning they represent the same information type and permit vertical concatenation (row appending) @xcite . as a top-k retrieval task, tus ranks candidate tables c by a table-level relevance score r(q, c). this score is typically obtained by aggregating column-level semantic relevance scores r(c q , c c ) computed for each column c q of the query table q and each column c c of the candidate table c. the aggregation often involves finding an optimal mapping between the columns of q and c, for instance via maximum bipartite matching @xcite . successful tus facilitates data integration and dataset enrichment @xcite . recent research has introduced sophisticated tus methods with complex representation learning @xcite @xcite designed to capture deeper semantics. however, current benchmarks often exhibit excessive schema overlap, limited semantic complexity, and potential ground truth inconsistencies, which raises questions about whether they provide a reliable environment to evaluate advanced tus capabilities. while state-of-the-art methodologies leverage semantic reasoning to reflect the task specific challenges, observed high performance may be significantly attributed to model adaptation to specific statistical and structural properties inherent within the benchmark datasets. this phenomenon can confound the accurate assessment and potentially underestimate the isolated contribution of improvements specifically targeting semantics-aware tus. in this paper, we examine prominent tus benchmarks foot_0 , using simple baselines to assess the benchmarks themselves. our research questions are: our analysis foot_1 reveals that simple baseline methods often achieve surprisingly strong performance by leveraging benchmark characteristics rather than demonstrating sophisticated semantic reasoning. our contributions include: we review existing research on tus methods and the benchmarks used for their evaluation, with a focus on how underlying assumptions about table unionability have evolved to become increasingly nuanced and complex. ## methodology as tus methods become increasingly sophisticated, the benchmarks used for their evaluation may contain inherent characteristics that hinder the accurate assessment of progress in semantic understanding. this section outlines our approach to examining prominent tus benchmarks through analysis of their construction methods and strategic use of simple baselines as diagnostic tools. the goal of advanced tus methods is to capture deep semantic compatibility between tables, beyond simple lexical or structural similarity. our investigation first analyzes the various benchmark construction processes to identify potential structural weaknesses, then employs computationally inexpensive baseline methods to reveal how these characteristics enable alternative pathways to high performance, thereby influencing evaluation outcomes. ## experimental setup to evaluate our hypotheses about benchmark limitations, we employ both simple baseline methods (section 3.2) and advanced sota methods in a controlled experimental framework. this section details the benchmark datasets used, any necessary preprocessing, the comparative methods, and our standardized evaluation approach. ## results and discussion our empirical evaluation revealed significant patterns across benchmarks that expose fundamental limitations in their ability to measure progress in semantic understanding. tables 2 and 3 present effectiveness and efficiency metrics respectively. ## towards better tus benchmarks in industry practice, unionability judgments are inherently subjective, depending on analytical goals, domain contexts, data accessibility constraints @xcite , and user preferences @xcite . yet current benchmarks impose fixed definitions, creating a disconnect with practical utility: methods excelling on benchmarks often falter in real-world scenarios demanding different compatibility thresholds. addressing this requires benchmark designs that embrace contextual variability and provide a stable foundation for evaluation, lest even advanced methods fall short in practice. ## conclusion our analysis of tus benchmarks highlights three major limitations: excessive overlap in partitioningbased datasets, semantics easily captured by pretrained embeddings, and non-negligible groundtruth inconsistencies. the first two allow simple baselines to rival sophisticated models with far lower computational cost, showing that high performance isn't necessarily tied to advanced semantic reasoning. the third undermines evaluation validity, as scores may reflect misalignment with flawed ground truth rather than actual benchmark difficulty. this gap between benchmark performance and true semantic capability suggests current evaluations often reward adaptation to benchmarkspecific artifacts. to address this, we propose design principles that better reflect the complex, subjective nature of real-world table union search. limitations: our study examined selected benchmarks and methods, with broader evaluation potentially revealing more insight. our investigation of ground truth issues in ugen and lakebench, while systematic, identifies certain patterns without exhaustive quantification. future work: developing benchmarks aligned with our proposed criteria represents the next step towards ensuring that measured progress translates to meaningful real-world utility.