ymoslem/acl-abstracts huggingface dataset:
OSU CompLing at the GEM`24 Data-to-Text Task (2024)

This paper details experiments conducted for completing the GEM 2024 Data-to-Text task for a WebNLG dataset (Gardent et al., 2017). We show that model performance varies greatly across English, Spanish, Chinese, and Russian. Data filtering was done with automatic model judgments via error detection, which performs differently per language. We report English and Spanish dev set results for a data filtering and knowledge distillation approach to generating natural language outputs for sets of triples across a variety of domains. Specifically, we compare three generation conditions: 1) few-shot prompting with ChatGPT (GPT4), 2) fine-tuning LLama2 on the unfiltered dataset, and 3) fine-tuning Llama2 on a filtered version of the dataset. Russian and Chinese efforts did not result in submissions due to inconsistent or incoherent translations being produced in either the data synthesis or final generation stages. We provide details on these shortcomings but largely focus on Spanish and English efforts that align with our task submissions. We ultimately submitted outputs in English and Spanish that were generated using a version of Llama2 fine-tuned on a filtered dataset.
CUET\_SSTM at the GEM`24 Summarization Task: Integration of extractive and abstractive method for long text summarization in Swahili language (2024)

Swahili, spoken by around 200 million people primarily in Tanzania and Kenya, has been the focus of our research for the GEM Shared Task at INLG`24 on Underrepresented Language Summarization. We have utilized the XLSUM dataset and have manually summarized 1000 texts from a Swahili news classification dataset. To achieve the desired results, we have tested abstractive summarizers (mT5{\_}multilingual{\_}XLSum, t5-small, mBART-50), and an extractive summarizer (based on PageRank algorithm). But our adopted model consists of an integrated extractive-abstractive model combining the Bert Extractive Summarizer with some abstractive summarizers (t5-small, mBART-50). The integrated model overcome the drawbacks of both the extractive and abstractive summarization system and utilizes the benefit from both of it. Extractive summarizer shorten the paragraphs exceeding 512 tokens, ensuring no important information has been lost before applying the abstractive models. The abstractive summarizer use its pretrained knowledge and ensure to generate context based summary.
The LSG Challenge Workshop at INLG 2024: Prompting Techniques for Crafting Extended Narratives with LLMs (2024)

The task of generating long narratives using Large Language Models (LLMs) is a largely unexplored area within natural language processing (NLP). Although modern LLMs can handle up to 1 million tokens, ensuring coherence and control over long story generation is still a significant challenge. This paper investigates the use of summarization techniques to create extended narratives, specifically targeting long stories. We propose a special prompting scheme that segments the narrative into several parts and chapters, each generated iteratively with contextual information. Our approach is evaluated with GAPELMAPER, a sophisticated text coherence metric, for automatic evaluation to maintain the structural integrity of the generated stories. We also rely on human evaluation to assess the quality of the generated text. This research advances the development of tools for long story generation in NLP, highlighting both the potential and current limitations of LLMs in this field.
A Report on LSG 2024: LLM Fine-Tuning for Fictional Stories Generation (2024)

Our methodology centers around fine-tuning a large language model (LLM), leveraging supervised learning to produce fictional text. Our model was trained on a dataset crafted from a collection of public domain books sourced from Project Gutenberg, which underwent thorough processing. The final fictional text was generated in response to a set of prompts provided in the baseline. Our approach was evaluated using a combination of automatic and human assessments, ensuring a comprehensive evaluation of our model`s performance.
Sumotosima : A Framework and Dataset for Classifying and Summarizing Otoscopic Images (2024)

Otoscopy is a diagnostic procedure to examine the ear canal and eardrum using an otoscope. It identifies conditions like infections, foreign bodies, eardrum perforations, and ear abnormalities. We propose a novel resource-efficient deep learning and transformer-based framework, Sumotosima (Summarizer for Otoscopic Images), which provides an end-to-end pipeline for classification followed by summarization. Our framework utilizes a combination of triplet and cross-entropy losses. Additionally, we use Knowledge Enhanced Multimodal BART, where the input is fused textual and image embeddings. The objective is to deliver summaries that are well-suited for patients, ensuring clarity and efficiency in understanding otoscopic images. Given the lack of existing datasets, we have curated our own OCASD (Otoscopy Classification And Summary Dataset), which includes 500 images with 5 unique categories, annotated with their class and summaries by otolaryngologists. Sumotosima achieved a result of 98.03{\%}, which is 7.00{\%}, 3.10{\%}, and 3.01{\%} higher than K-Nearest Neighbors, Random Forest, and Support Vector Machines, respectively, in classification tasks. For summarization, Sumotosima outperformed GPT-4o and LLaVA by 88.53{\%} and 107.57{\%} in ROUGE scores, respectively. We have made our code and dataset publicly available at https://github.com/anas2908/Sumotosima
Exploring Expected Answer Types for Effective Question Answering Systems for low resource language (2024)

Question-answering (QA) systems play a pivotal role in natural language processing (NLP), powering applications such as search engines and virtual assistants by providing accurate responses to user queries. However, building effective QA systems for Dravidian languages, like Tamil, poses distinct challenges due to the scarcity of resources and the linguistic complexities inherent to these languages. This paper introduces a novel method to enhance QA accuracy by integrating answer-type features alongside traditional question and context inputs. We fine-tuned both mono- and multilingual pre-trained models on the Extended Chaii dataset, which comprises Tamil translations from the SQuAD dataset, as well as on the SQuAD-EAT-5000 dataset, consisting of English-language instances. Our experiments reveal that incorporating answer-type features significantly improves model performance compared to using only question and context inputs. Specifically, for the Extended Chaii dataset, the MuRIL model achieved the highest F1 score of 53.89, surpassing other pre-trained models, while RoBERTa outperformed BERT on the SQuAD-EAT-5000 dataset with a score of 82.07. This research advances QA systems for Dravidian languages and underscores the importance of integrating linguistic features for improved accuracy.
Precision Empowers, Excess Distracts: Visual Question Answering With Dynamically Infused Knowledge In Language Models (2024)

In the realm of multimodal tasks, Visual Question Answering (VQA) plays a crucial role by addressing natural language questions grounded in visual content. Knowledge-Based Visual Question Answering (KBVQA) advances this concept by adding external knowledge along with images to respond to questions. We introduce an approach for KBVQA, augmenting the existing vision-language transformer encoder-decoder (OFA) model . Our main contribution involves enhancing questions by incorporating relevant external knowledge extracted from knowledge graphs, using a dynamic triple extraction
Improving Few-shot Prompting using Cluster-based Sample Retrieval for Medical NER in Clinical Text (2024)

Named Entity Recognition (NER) in the medical domain is crucial for extracting essential information from clinical text. Large Language Models (LLMs) have demonstrated remarkable capabilities in this task, but their performance is highly dependent on the quality of the prompts. Few-shot prompting or prompt-by-example, where the input query to LLM is augmented with one or more sample outputs, is a well-known technique in guiding the LLMs to the expected result. The quality of the sample in the prompt plays an important role in this task. This paper proposes to improve the performance of few-shot prompting for medical NER on clinical text using a cluster-based strategy for sample selection. We employ the concepts from Retrieval Augmented Generation (RAG) and K-means clustering to identify the most similar annotated examples for any given input text. Using these contextually relevant yet divergent training samples as examples, we guide the LLM toward extracting more accurate medical entities. Our experiments using the llama-2 model show that this approach significantly outperforms zero-shot prompting and random sampled few-shot prompting in two data sets chosen for this study, demonstrating the efficacy of cluster-based retrieval in improving few-shot prompting for medical NER tasks.
Standardizing Genomic Reports: A Dataset, A Standardized Format, and A Prompt-Based Technique for Structured Data Extraction (2024)

Extracting information from genomic reports of cancer patients is crucial for both healthcare professionals and cancer research. While Large Language Models (LLMs) have shown promise in extracting information, their potential for handling genomic reports remains unexplored. These reports are complex, multi-page documents that feature a variety of visually rich, structured layouts and contain many domain-specific terms. Two primary challenges complicate the process: (i) extracting data from PDFs with intricate layouts and domain-specific terminology and (ii) dealing with variations in report layouts from different laboratories, making extraction layout-dependent and posing challenges for subsequent data processing. To tackle these issues, we propose GR-PROMPT, a prompt-based technique, and GR-FORMAT, a standardized format. Together, these two convert a genomic report in PDF format into GR-FORMAT as a JSON file using a multimodal LLM. To address the lack of available datasets for this task, we introduce GR-DATASET, a synthetic collection of 100 cancer genomic reports in PDF format. Each report is accompanied by key-value information presented in a layout-specific format, as well as structured key-value information in GR-FORMAT. This is the first dataset in this domain to promote further research for the task. We performed our experiment on this dataset.
Identification of Idiomatic Expressions in Konkani Language Using Neural Networks (2024)

The task of multi-word expressions identification and processing has posed a remarkable challenge to the natural language processing applications. One related subtask in this arena is correct labelling of the sentences with the presence of idiomatic expressions as either literal or idiomatic sense. The regional Indian language Konkani spoken in the states located in the west coast of India lacks in the research in idiom processing tasks. We aim at bridging this gap through a contribution to idiom identification method in Konkani language. This paper classifies the idiomatic expression usage in Konkani language as idiomatic or literal usage using a neural network-based setup. The developed system was able to successfully perform the identification task with an accuracy of 79.5 {\%} and F1-score of 0.77.
Konkani Wordnet Visualizer as a Concept Teaching-Learning Tool (2024)

The Visualizer is a tree-structure designed to browse and explore the Konkani WordNet lexical database. We propose to utilise this tool as a concept teaching and learning resource for Konkani, to be used by both teachers and students. It can also be used to add the missing semantic and lexical relations, thus enhancing the wordnet. It extracts related concepts for a given word and displays them as a sub-tree. The interface includes various features to offer users greater flexibility in navigating and understanding the word relationships. We attempted to enrich the Konkani Wordnet qualitatively with a Visualizer that offers an improved usability and is incorporated in the Konkani Wordnet website for the public use. The Visualizer is designed to provide graphical representations of words and their semantic relationships, making it easier to explore connections and meanings within the lexical database.
A Systematic Exploration of Linguistic Phenomena in Spoken Hindi: Resource Creation and Hypothesis Testing (2024)

This paper presents a meticulous and well-structured approach to annotating a corpus of Hindi spoken data. We deployed 4 annotators to augment the spoken section of the EMILLE Hindi corpus by marking the various linguistic phenomena observed in spoken data. Then we analyzed various phonological (sound deletion), morphological (code-mixing and reduplication) and syntactic phenomena (case markers and ambiguity), not attested in written data. Code mixing and switching and constitute the majority of the phenomena we annotated, followed by orthographic errors related to symbols in the Devanagiri script. In terms of divergences from written form of Hindi, case marker usage, missing auxiliary verbs and agreement patterns are markedly distinct for spoken Hindi. The annotators also assigned a quality rating to each sentence in the corpus. Our analysis of the quality ratings revealed that most of the sentences in the spoken data corpus are of moderate to high quality. Female speakers produced a greater percentage of high quality sentences compared to their male counterparts. While previous efforts in corpus annotation have been largely focused on creating resources for engineering applications, we illustrate the utility of our dataset for scientific hypothesis testing. Inspired from the Surprisal Theory of language comprehension, we validate the hypothesis that sentences with high values of lexical surprisal are rated low in terms of quality by native speakers, even when controlling for sentence length and word frequencies in a sentence.
A Corpus of Hindi-English Code-Mixed Posts to Hate Speech Detection (2024)

Social media content, such as blog posts, comments, and tweets, often contains offensive language, including racial hate speech, personal attacks, and sexual harassment. Detecting inappropriate language is crucial for user safety and prevention of hateful behavior and aggression. This study introduces the HECM (Hindi-English code-mixed tweets) to fill the gap in Hindi language resources. The corpus comprises approximately 9.4K tweets labeled as hateful and nonhateful. It includes detailed information on the data, such as the annotation schema, the label definitions, and an interannotator agreement score of 85{\%}. The study evaluates the effectiveness of traditional machine learning, deep neural networks, and transformer encoder-based approaches. The results show a significant improvement in terms of macro-F1 and weighted F1 scores. Additionally, a lexicon containing 2000 lexicons tagged in 21 categories is created based on the multilingual HURTLEX lexicon. This lexicon is merged with the transformer encoder, resulting in a marginal improvement in macro-F1 and weighted-F1. The study also experiments with a Hindi-Devanagari dataset to assess the impact of the lexicon on performance metrics.
An Aid to Assamese Language Processing by Constructing an Offline Assamese Handwritten Dataset (2024)

Recent years have seen a growing interest in analyzing Indian handwritten documents. In pattern recognition, particularly handwritten document recognition, the availability of standard databases is essential for assessing algorithm efficacy and facilitating result comparisons among research groups. However, there is a notable scarcity of standardized databases for handwritten texts in Indian languages. This paper presents a comprehensive methodology for the development of a novel, unconstrained dataset named OAHTD (Offline Assamese Handwritten Text Dataset) for the Assamese language, derived from offline handwritten documents. The dataset, which represents a significant contribution to the field of Optical Character Recognition (OCR) for handwritten Assamese, is the first of its kind in this domain. The corpus comprises 410 document images, each containing a diverse array of linguistic elements including words, numerals, individual characters, and various symbols. These documents were collected from a demographically diverse cohort of 300 contributors, spanning an age range of 10 to 76 years and representing varied educational backgrounds and genders. This meticulously curated collection aims to provide a robust foundation for the development and evaluation of OCR algorithms specifically tailored to the Assamese script, addressing a critical gap in the existing literature and resources for this language.
Monolingual text summarization for Indic Languages using LLMs (2024)

We have analyzed the growth of advanced text summarization method leveraging LLM for Indic language. Text summarization involves transforming a longer text information into a more concise version, ensuring that the most prominent information and key meanings are maintained. Our goal is to produce concise and accurate summaries from longer texts, focusing on maintaining detailed information and coherence. We utilize NLP techniques for text cleaning, keyword extraction and summarization, along with performance evaluation metrics such as ROUGE score, BLEU score and BERT Score. The results demonstrate an incremental improvement in the quality of generated summaries, with a particular emphasis on enhancing informativeness while minimizing redundancy. This research work also highlights the importance of tuning parameters and leveraging advanced models for producing high quality summaries in diverse domains for Indic Language.
Enhancing Trust and Interpretability in Malayalam Sentiment Analysis with Explainable AI (2024)

Natural language processing (NLP) has seen a rise in the use of explainable AI, especially for low-resource languages like Malayalam. This study builds on our earlier research on sentiment analysis which uses identified views to classify and understand the context. Support Vector Machine (SVM) and Random Forest (RF) classifiers are two machine learning approaches that we used to do sentiment analysis on the Kerala political opinion corpus. Using Bag-of-Words (BoW) and Term Frequency-Inverse Document Frequency (TF-IDF) features, we construct feature vectors for sentiment analysis. In this, analysis of the Random Forest classifier`s performance shows that it outperforms SVM in terms of accuracy and efficiency, with an accuracy of 85.07 {\%}. Using Local Interpretable Model-Agnostic Explanations (LIME) as a foundation, we address the interpretability of text classification and sentiment analysis models. This integration increases user confidence and model use by offering concise and understandable justifications for model predictions. The study lays the groundwork for future developments in the area by demonstrating the significance of explainable AI in NLP for low-resource languages.
Automatic Sanskrit Poetry Classification Based on K\=avyaguṇa (2024)

K{\={a}}vyaguṇa denotes the syntactic and phonetic attributes or qualities of Sanskrit poetry that enhance its artistic appeal, commonly classified into three categories: M{\={a}}dhyurya (Sweetness), Oja (Floridity), and Pras{\={a}}da (Lucidity). This paper presents the K{\={a}}vyaguṇa Classifier, a machine learning module, designed to classify Sanskrit literary texts into three distinct guṇas, by employing a diverse range of machine learning algorithms, including Random Forest, Gradient Boosting, XGBoost, Multi-Layer Perceptron and Support Vector Machine. For vectorization, we employed two methods: the neural network-based Word2vec and a custom feature engineering approach grounded in the theoretical understanding of K{\={a}}vyaguṇas as described in Sanskrit poetics. The feature engineering model significantly outperformed, achieving an accuracy of up to 90.6{\%}
From Data to Insights: The Power of LM`s in Match Summarization (2024)

The application of Natural Language Processing is progressively extending into many domains as time progresses. We are motivated to evaluate language model`s (LMs) capabilities in many real-world domains due to their significant potential. This study examines the use of LMs in sports, explicitly emphasizing their ability to convert data into text and their understanding of cricket. By examining cricket scorecards, a widely played sport on the Indian subcontinent and many other regions, we will evaluate the summaries produced by LMs from several viewpoints. We have collected concise summaries of the scorecards from the ODI World Cup 2023 and assessed them in terms of both factual accuracy and sports-specific significance. We analyze the specific factors that are included in the summaries and those that are excluded. Additionally, it analyzes prevalent mistakes concerning completeness, correctness, and conciseness. We are presenting our findings here and also our dataset and code are available https://github.com/satyawork/ODI-WORLDCUP.git
Sentiment and sarcasm: Analyzing gender bias in sports through social media with deep learning (2024)

Gender bias continues to be a pervasive issue, especially in public discourse surrounding high-profile events like the Olympics. Social media platforms, particularly Twitter, have become a central space for discussing such biases, making it crucial to analyze these conversations to better understand public attitudes. Sentiment analysis plays a key role in this effort by determining how people feel about gender bias. However, sarcasm often complicates sentiment analysis by distorting the true sentiment of a tweet, as sarcastic expressions can mask negative or positive sentiments. To address this, the study introduces a novel framework called SENSA (SENtiment and Sarcasm Analysis), designed to detect both sentiment and sarcasm in tweets related to gender bias. The framework leverages the R2B-CNN model for robust sarcasm and sentiment classification. Using approximately 5,000 tweets related to gender bias from 2010 to August 30, 2024, SENSA applies advanced sarcasm detection to account for shifts in sentiment caused by sarcastic remarks. The R2B-CNN model demonstrates a high accuracy of 92.32{\%} along with achieving 92.75{\%} precision and 92.53{\%} F1-score for sarcasm detection and a 93.67{\%} accuracy, 92.33{\%} precision and 92.33{\%} F1-score for sentiment classification. SENSA provides a comprehensive understanding of gender bias discussions on social media by capturing both sentiment and sarcasm to reveal deeper insights into public perceptions.
Quality Estimation of Machine Translated Texts based on Direct Evidence Approach (2024)

Quality Estimation task deals with the estimation of quality of translations produced by a Machine Translation system without depending on Reference Translations. A number of approaches have been suggested over the years. In this paper we show that the parallel corpus used as training data for training the MT system holds direct clues for estimating the quality of translations produced by the MT system. Our experiments show that this simple, direct and computationally efficient method holds promise for quality estimation of translations produced by any purely data driven machine translation system.
Reconsidering SMT Over NMT for Closely Related Languages: A Case Study of Persian-Hindi Pair (2024)

This paper demonstrates that Phrase-Based Statistical Machine Translation (PBSMT) can outperform Transformer-based Neural Machine Translation (NMT) in moderate-resource scenarios, specifically for structurally similar languages, Persian-Hindi pair in our case. Despite the Transformer architecture`s typical preference for large parallel corpora, our results show that PBSMT achieves a BLEU score of 66.32, significantly exceeding the Transformer-NMT score of 53.7 ingesting the same dataset.