Evaluating Structural and Linguistic Quality in U rdu DRS Parsing and Generation through Bidirectional Evaluation (2025)

Abstract: Evaluating Discourse Representation Structure (DRS)-based systems for semantic parsing (Text-to-DRS) and generation (DRS-to-Text) poses unique challenges, particularly in low-resource languages like Urdu. Traditional metrics often fall short, focusing either on structural accuracy or linguistic quality, but rarely capturing both. To address this limitation, we introduce two complementary evaluation methodologies—Parse-Generate (PARS-GEN) and Generate-Parse (GEN-PARS)—designed for a more comprehensive assessment of DRS-based systems. PARS-GEN evaluates the parsing process by converting DRS outputs back to the text, revealing linguistic nuances often missed by structure-focused metrics like SMATCH. Conversely, GEN-PARS assesses text generation by converting generated text into DRS, providing a semantic perspective that complements surface-level metrics such as BLEU, METEOR, and BERTScore. Using the Parallel Meaning Bank (PMB) dataset, we demonstrate our methodology across Urdu, uncovering unique insights into Urdu’s structural and linguistic interplay. Findings show that traditional metrics frequently overlook the complexity of linguistic and semantic fidelity, especially in low-resource languages. Our dual approach offers a robust framework for evaluating DRS-based systems, enhancing semantic parsing and text generation quality.

## introduction drs is central to advanced semantic processing, providing a flexible and language-neutral framework for capturing complex semantic nuances beyond basic text interpretation @xcite , including phenomena such as negation and quantification @xcite . its adaptability makes drs ideal for multilingual natural language processing (nlp) systems, offering a unified way of representing meaning across languages with diverse structural and syntactic properties @xcite . drs parsing @xcite @xcite and generation @xcite @xcite are reversible processes which pose unique challenges, especially when working with urdu-a morphologically rich language. urdu exhibits different syntactic structures and semantic expressions, making accurate evaluation difficult due to the limitations of traditional structural and surface-level metrics @xcite . existing evaluations often fail to fully account for linguistic and structural accuracy across languages, which is essential for ensuring meaningful cross-linguistic semantic representation. this gap has motivated our development of innovative evaluation methods to bridge structural precision with linguistic adequacy in drs-based systems. our research primarily aims to create evaluation frameworks that integrate both structural and linguistic (in the sense of surface-level) assessments. to accomplish this, we introduce two bidirectional evaluation paradigms-pars/pars-gen and gen/gen-pars. the former assesses parsing quality by examining the linguistic coherence of the text generated from drs structures, moving beyond traditional metrics to provide insights into how well structural accuracy supports meaningful language representation. conversely, the latter evaluates generation quality by analyzing the semantic consistency of parsed structures derived from generated text, offering a deeper perspective than surface-level comparisons alone. semantic parsing evaluation typically relies on structural metrics like smatch @xcite , which assesses roles or concepts-based overlaps between predicted and reference drs text generation from drs also poses a unique evaluation challenge. traditional metrics like bleu @xcite , meteor @xcite , and even recent metrics like bertscore @xcite prioritize surface-level similarities between generated and reference texts. however, given the diversity of natural language, there can be multiple valid expressions for the same meaning. for example, the sentences ("john gave mary some money") and ("john gave the money to mary") convey similar meanings, but their syntactic variations lead to low scores under traditional evaluations, despite perfect semantic equivalence in drs representation (ex. 4, in this context, our research investigates several critical questions: (i) how can the evaluation of semantic parsing and text generation be improved beyond existing structural and surface-level metrics? (ii) how does structural accuracy in semantic parsing influence linguistic quality in text gen-eration? (iii) how can surface-level evaluations be enhanced by assessing individual lexical entities? (iv) can the reversible nature of semantic parsing and text generation be exploited for improved evaluations? and (v) do these alternate evaluations correlate with each other and are they statistically significant? to address these questions, this paper makes the following key contributions: (i) it introduces novel evaluation paradigms, pars/pars-gen and gen/gen-pars, which reveal unique insights into the language's syntactic variability and complex semantic structures that traditional metrics often overlook; (ii) the pars/pars-gen paradigm uses linearized text to mitigate nonoptimal outcomes in smatch's greedy search algorithm, enabling a more intuitive and humancentered approach to parsing evaluation; (iii) through the gen/gen-pars evaluation, it identifies semantic and syntactic issues at a node level, examining lexical drs concepts like nouns, verbs, adjectives, and adverbs within the generated drs to provide a granular view of the generation quality, ultimately facilitating a balanced metric that captures both structural and linguistic fidelity; and (iv) it proposes a detailed pearson correlation analysis between pars/pars-gen, gen/gen-pars. the observed statistically significant correlations underscore the robustness of our approach and demonstrate the effectiveness of combining structural and linguistic assessments in drs-based semantic processing. the remaining sections are organized as follows: section 2 discusses the limitations of current evaluation approaches in detail. section 3 presents our novel evaluation methodologies and describes the experimental setup and implementation details. section 4 presents reversible evaluation measures and correlation analysis. finally, section 5 concludes with limitations. ## limitations in current evaluations the evaluation of semantic parsing and text generation system presents unique challenges that conventional metrics often struggle to address comprehensively. this section examines these limitations in detail and establishes the motivation for our proposed evaluation approaches. parsing limitations: traditional evaluation metrics, such as smatch @xcite , smatch++ @xcite , and sembleu @xcite , focus on assessing structural similarities between predicted and reference drs. smatch, for instance, employs a greedy hillclimbing algorithm that matches nodes across logical structures. this approach, however, often results in suboptimal evaluations, especially in cases where structural differences do not reflect actual semantic deviations. for example, smatch assigns a zero score to the drs representation for ("tom bought a new pickup"), despite the semantic content being essentially equivalent in both gold and predicted drs. the low-score is due to minor structural differences, underscoring a limita-tion of smatch's focus on structural alignment rather than semantic equivalence (ex. 1, additionally, smatch's handling of semantic relationships is limited, as it treats drs nodes as isolated entities. this limitation is evident in ex. 2 ("tom shows mary a picture of his dog"), where differences in role modifiers like ("topic" and "creator") for "picture" results in a smatch score of 69.23. the metric's penalty for these isolated structural variations, without accounting for the underlying semantic alignment, highlights its tendency to overlook contextually equivalent expressions when modifiers are altered or substituted. this penalization is further illustrated in ex. 3 ("today i have a pain in my neck"), where smatch deducts points based on minor discrepancies in the verb sense, yielding a score of 60.00 despite the overall message being well-preserved across both drs. in ex. 4 ("thirteen people were arrested"), smatch once again assigns a score of zero, this time due to an inconsistency in the numerical value between gold (13) and predicted (30) drs. this significant deduction overlooks that the core event-people being arrested-is accurately conveyed. ex. 5 ("i have a lot of money") further emphasizes smatch's limitations, where minor numerical and role-discrepancies lead to a score of 57.89, despite the intended meaning being largely retained. these examples collectively underscore that smatch's sensitivity to structural changes can cause unfairly low scores even when semantic content is mostly preserved. to address these limitations, our pars-gen approach rephrases drs outputs as natural language text, enabling the use of complementary evaluation metrics like chrf, meteor, and bertscore, which emphasizes semantic accuracy. by generating interpretable text from drs, pars-gen provides a holistic evaluation of parsing quality and captures linguistic nuances that structural metrics like smatch often miss. through this approach, we enhance the accessibility and interpretability of semantic fidelity assessment, ensuring a more accurate and inclusive evaluation across diverse language structures and semantics. generation limitations: traditional evaluation metrics for urdu text generation, like bleu, me-teor, and rouge, primarily rely on n-gram overlaps, limiting their ability to capture semantic alignment beyond lexical matches. this is particularly evident in our drs-to-text generation examples in bertscore, which attempts to measure seman-tic similarity using pre-trained language models, is better suited for capturing the deeper semantic relationships between words. however, even this metric struggles when dealing with syntactic and morphological variations in urdu. for instance, in ex. 3, "everyone left" and "all have now left" exhibit a difference in tense and aspect, yet the meaning remains intact. bertscore performs better here with scores of 88.74, but still faces challenges when evaluating minor syntactic differences that do not affect the overall meaning. these examples underscore the need for an evaluation approach that emphasizes semantic quality. our gen-pars approach addresses this by focusing on whether generated texts preserve the semantic content of the original drs. across all examples analyzed, where traditional metrics like bleu fluctuate significantly (ranging from 16.67 to 56.43), gen-pars achieves perfect smatch scores of 1.0 by parsing generated texts back to drs. this validates semantic equivalence despite surface differences as evident in ## methods and results this study presents two novel evaluation methodologies for assessing the quality of drs parsing and generation in urdu: (1) evaluating parsing through generation capabilities (pars-gen) and (2) assessing generation through semantic parsing (gen-pars). unlike conventional metrics that often focus on surface-level text similarity or structural alignment, these methodologies offer a deeper, cross-task approach that assesses both structural and linguistic fidelity in urdu semantic processing. to complement our cross-task evaluations, we also computed the pearson correlation between metrics across the pars/pars-gen and gen/gen-pars evaluations. this correlation analysis helps us understand the relationship between structural accuracy (e.g., smatch f1 scores) and linguistic quality metrics (e.g., bleu, meteor, bertscore). pmb foot_1 is a multilingual dataset comprising semantic representations in english, italian, german, dutch, and chinese. leveraging the language-neutral nature of drs, we transformed english drs-text pairs into urdu through a sys-tematic approach involving syntactic structure, concept and word alignment, grammatical genders, and cross-lingual adaptation through named entities. this methodology resulted in the first comprehensive semantic resource for urdu, comprising 3,000 gold-standard (fully manually annotated) data instances. the dataset transformation employed a hybrid methodology: drs transformations utilized rule-based techniques and human annotation, while text translations were generated using google translate api. the dataset was partitioned into 1,200 training, 900 development, and 900 test examples. to enhance dataset diversity and complexity, we applied multi-dimensional augmentation strategies, including named entities, lexical (encompassing common nouns, adjectives, adverbs, and verbs), and grammatical augmentations. this approach expanded the dataset to 10,800 training examples, supplemented by 6,857 silver (partially manually annotated) instances. for bidirectional evaluation-converting pars to pars-gen and vice versa-we employed byt5-based parsing and generation models, fine-tuned using our comprehensive augmented dataset foot_2 . we implemented a two-stage fine-tuning strategy consistent with @xcite . the first stage involved fine-tuning the model on silver data for 3 epochs to establish foundational drs knowledge. the second stage focused on gold data fine-tuning for 10 epochs. experimental parameters included adamw optimizer, polynomial learning rate decay (1e-4), batch size of 32, maximum sequence length of 512, and geglu activation function. these models achieved stateof-the-art performance in urdu drs processing, facilitating reversible data generation. ## analysis and discussion to further emphasize the usefulness of the reversible evaluation approaches, we have analyzed examples present in reversible evaluation measures: while sections 2 highlighted the limitations of traditional parsing and generation metrics individually, in this section we present the cases where our proposed evaluation approaches (pars-gen and gen-pars) provide complementary evidence of semantic and structural preservation. through detailed analysis, we demonstrate how low scores in one type of evaluation (pars or gen) can be counter-verified by evaluating it in the reverse direction, revealing semantic equivalences that would have been missed. ## conclusion drs parsing and generation are reversible processes that can be exploited in cross-task evaluations. traditional metrics often fall short in capturing the true structural and linguistic quality required for accurate assessment. to address the limitations, we introduced two complementary methodologies, pars-gen and gen-pars, which offer a bidirectional framework to evaluate urdu drs processing more holistically. the pars-gen approach assesses parsing quality by generating text from parsed drs, revealing linguistic nuances that purely structural metrics may miss. in parallel, gen-pars transforms generated text back into drs, providing a structural and semantic evaluation of generation quality that goes beyond surface evaluations. applying these methods to urdu has yielded significant insights: (i) urdu exhibits stronger correlations between generation quality and parsing accuracy than the reverse, indicating that high-quality generation is a reliable predictor of parsing performance; (ii) bertscore shows the highest correlations, demonstrating their effectiveness in capturing urdu's complex linguistic features; and (iii) the positive, statistically significant correlations across both evaluation directions validate the bidirectional parsing-generation relationship for urdu.