H indi Reading Comprehension: Do Large Language Models Exhibit Semantic Understanding? (2025)

Abstract: In this study, we explore the performance of four advanced Generative AI models—GPT-3.5, GPT-4, Llama3, and HindiGPT, for the Hindi reading comprehension task. Using a zero-shot, instruction-based prompting strategy, we assess model responses through a comprehensive triple evaluation framework using the HindiRC dataset. Our framework combines (1) automatic evaluation using ROUGE, BLEU, BLEURT, METEOR, and Cosine Similarity; (2) rating-based assessments focussing on correctness, comprehension depth, and informativeness; and (3) preference-based selection to identify the best responses. Human ratings indicate that GPT-4 outperforms the other LLMs on all parameters, followed by HindiGPT, GPT-3.5, and then Llama3. Preference-based evaluation similarly placed GPT-4 (80%) as the best model, followed by HindiGPT(74%). However, automatic evaluation showed GPT-4 to be the lowest performer on n-gram metrics, yet the best performer on semantic metrics, suggesting it captures deeper meaning and semantic alignment over direct lexical overlap, which aligns with its strong human evaluation scores. This study also highlights that even though the models mostly address literal factual recall questions with high precision, they still face the challenge of specificity and interpretive bias at times.

## introduction machine reading comprehension (mrc) in natural language processing (nlp) is the task of making machines retrieve or generate precise and contextually relevant answers from a specific question and a body of text @xcite . it has numerous real-world applications, ranging from search engines to educational tools and domain-specific con1 human annotations available at https://github.com/ dml2611/hindimrc . versational agents or chatbots @xcite . mrc involves understanding the underlying context and is extremely challenging as it requires complex cognitive capabilities like summarising, sequencing, inferencing, and comparing and contrasting facts presented in the given text @xcite . while nlp has seen significant advancements for widely spoken languages, much of the research has left lowresource languages like hindi underexplored, especially for complex tasks such as mrc. @xcite . hindi, the fourth most-spoken language globally @xcite , has witnessed major breakthroughs in nlp technologies in recent years. nevertheless, as large language models (llms) emerge as the cornerstone of nlp research, it is essential to ask: how well do these models understand hindi? while llms perform admirably on surface-level tasks like text generation, text classification, and machine translation @xcite that do not always require in-depth analysis of comprehension; mrc, that involves nuanced understanding of context, factual information, and reasoning, can serve as a benchmark for assess-ing the comprehension abilities of these models for hindi texts. in this study, we investigate the performance of four prominent llms-gpt-3.5 @xcite , @xcite , hindigpt 2 , and llama3 @xcite -to uncover how well these models perform on hindi reading comprehension tasks-not just in terms of accurate answers, but the limits of their comprehension and informativeness. to assess the performance of each model, we conducted both automatic and human evaluations (rating-based and preferencebased), as shown in figure 2 . the automatic evaluations provide a quantitative assessment of the models, while the human evaluations enable a qualitative assessment of each model's responses. this extensive study allows us to investigate and emphasize where these models thrive and where they fall short, as well as where they need to catch up to human comprehension. the rest of the paper is organized as follows. section 2 presents prior related work; section 3 outlines the methodology; section 4 states the results, followed by the conclusions and limitations in sections 5 and 5. ## related work researchers in the field of nlp consistently highlight the resource limitations that hinder the development of effective question-answering (qa) systems for low-resource languages such as hindi @xcite . the scarcity of high-quality, annotated datasets and linguistic tools specifically tailored for hindi is a significant barrier. state-of-the-art qa models, like bert and gpt, rely on extensive gold-standard corpora to produce accurate and robust results. however, for hindi, the availability of such resources remains limited, creating a gap in model performance @xcite . existing models and datasets are primarily designed for tasks involving short answer spans or multiple-choice responses, which restricts their flexibility. another significant challenge is the constrained context length used during model training, primarily due to computational costs associated with han-2 hindigpt available at https://chatgpt.com/g/ g-okgvbntmc-hindi-gpt dling large amounts of text @xcite . as a result, the models struggle to grasp the linguistic subtleties of hindi, such as syntax and morphology, which can reduce overall performance (ray et al., @xcite . the complexity of hindi is further increased by distinct syntactic structures, numerous semantic variants, and prevalent code-mixing (hindi-english hybrids) in written and spoken forms, add further barriers to qa development @xcite . however, llms have shown significant potential in handling diverse languages and can flexibly adapt to codemixed texts (brown, @xcite , making them potentially valuable tools to address hindi nlp challenges, like reading comprehension. ## methods in order to examine the comprehension abilities of gpt-3.5, gpt-4, hindigpt, and llama3 for hindi texts, the llms were directed to complete the hindi mrc task using instruction-based prompting on the hindirc foot_0 dataset, as described in sections 3.1 and 3.2, respectively. we evaluated the performances of the llms using both automatic and human assessments, as described in section 3.3. ## results the overall results of human and automatic evaluations, along with the inter-rater reliability, are covered in sections 4.1, 4.2, and 4.3, respectively. ## conclusion in this study, we use a novel triple assessment framework to compare the performance profiles of llms, gpt-3.5, gpt-4, llama3, and hindigpt for hindi reading comprehension. the ability of gpt-4 to generate contextually relevant and meaningful responses is demonstrated by its preference rating of 80%, which consistently outperformed competing models across all human-evaluated metrics-correctness, comprehension depth, and informativeness. with competitive scores, particularly in correctness and comprehension, hindigpt and gpt-3.5 trailed closely behind. gpt-3.5 was somewhat preferred above hindigpt for perceived understanding and precise responses. the results of automatic evaluations presented a contrasting picture, indicating fewer exact matches with reference texts, particularly for gpt-4 with lower n-gram metric scores (rouge and bleu). the high human evaluation scores of gpt-4 are consistent with its superior alignment with the un-derlying meaning as measured by semantic metrics (bleurt and cosine similarity), which show a greater grasp of the text than surface-level similarity. this comparison of automatic and human evaluations highlights the significance of semanticbased metrics and human evaluations for precisely assessing the level of a llm's comprehension, particularly in non-english languages like hindi.