SKV trio@ LT - EDI -2025: Hybrid TF - IDF and BERT Embeddings for Multilingual Homophobia and Transphobia Detection in Social Media Comments (2025)

Abstract: This paper presents a description of the paper submitted to the Shared Task on Homophobia and Transphobia Detection in Social Media Comments, LT-EDI at LDK 2025. We propose a hybrid approach to detect homophobic and transphobic content in low-resource languages using Term Frequency-Inverse Document Frequency (TF-IDF) and Bidirectional Encoder Representations from Transformers (BERT) for contextual embeddings. The TF-IDF helps capture the token’s importance, whereas BERT generates contextualized embeddings. This hybridization subsequently generates an embedding that contains statistical surface-level patterns and deep semantic understanding. The system uses principal component analysis (PCA) and a random forest classifier. The application of PCA converts a sparse, very high-dimensional embedding into a dense representation by keeping only the most relevant features. The model achieved robust performance across eight Indian languages, with the highest accuracy in Hindi. However, lower performance in Marathi highlights challenges in low-resource settings. Combining TF-IDF and BERT embeddings leads to better classification results, showing the benefits of integrating simple and complex language models. Limitations include potential feature redundancy and poor performance in languages with complex word forms, indicating a need for future adjustments to support multiple languages and address imbalances.

## introduction homophobia and transphobia are two concepts that foster negative opinions about homosexual and transgender individuals @xcite . these concepts are endemic in online communities and are most commonly expressed through context-dependent, subtle language that excludes lgbtq+ populations. as digital communication grows, automated systems to detect and mitigate negative content using discriminatory language become increasingly important. however, detecting such content in low-resource languages is a significant challenge posed by linguistic subtleties and the lack of available annotated data. this paper addresses the submission of the team skv trio to the shared task on homophobia and transphobia detection in social media comments @xcite . the proposed system was designed as a classification model with hybrid features. the proposed model combines features obtained from term frequency-inverse document frequency (tf-idf) and a pre-trained bidirectional encoder representations from transformers (bert) model. integrating these categories of features, the system achieves both surface patterns and deeper contextual meaning required for efficient classification. the classifier was trained using a random forest classifier with the combined feature representations. we employed principal component analysis (pca) to facilitate the reduction of dimensionality within the tf-idf embedding to match the tf-idf and bert embedding dimensionality. our experiments demonstrate that the fusion of tf-idf and bert embeddings significantly improves classification accuracy compared to using either feature extraction technique alone. the implementation is available at https://github.com/klaxmivignesh/skvtrio-/tree/main 2 related work the study @xcite to detect homophobia and transphobia makes the hypothesis that multilingual language models can detect hate speech on social media more effectively if data augmentation through pseudolabeling is used. to test the hypothesis, the study looked at a number of multilingual language models. with 5,193 malayalam and 3,203 hindi comments, @xcite offer a new, high-quality dataset for identifying homophobic and transphobic content in both languages. the dataset also contains tests on the malayalam, hindi, english, tamil, and tamil-english datasets using transformer-based deep learning models and conventional machine learning. to automatically identify homophobic and transphobic content, (chakravarthi, 2024) proposed a new dataset that was annotated by experts and trained machine learning models using it. 15,141 annotated comments in tamil, english, and both languages are included in the dataset. the average macro f1 score for the top tamil, english, and tamil-english systems was 0.570, 0.870, and 0.610, respectively. these scores were obtained by training a random forest with fasttext for tamil data and logistic regression with bert features for english and tamil-english data. @xcite proposed a new expert-labeled dataset for telugu, kannada, and gujarati, along with an expert-labeled dataset. the dataset includes extensive annotation rules, gathering about 10,000 comments in all three languages. a baseline model with pre-trained transformers was trained using the proposed dataset. the findings of a shared task to detect homophobic and transphobic language in social media posts are presented in @xcite and @xcite . task b required a fine-grained sevenclass classification, while task a classified comments into homophobic, transphobic, or neither categories. english, spanish, tamil, hindi, and malayalam were the five languages in which the task contained data. with the highest f1-scores of 0.997 for spanish in task a and 0.884 for tamil in task b, the top systems showed excellent performance. all the models reported in the literature are using either conventional techniques or bertbased models for generating features. bert-based models may not generate relevant features for rare words, which are important. these words can be efficiently captured by the tf-idf algorithm. ## dataset description the dataset is divided into training, development, and testing sets for each language. there is noticeable variation in dataset size across languages. kannada and telugu, for instance, have significantly more samples than hindi or tamil, introducing language imbalance that can ## methodology this section outlines the complete methodology of the system designed for this task. the methodology is divided into the following stages: data preprocessing, feature extraction, embedding fusion, model training, and prediction. figure 1 illustrates the machine learning pipeline we used for this shared task. three different data sources support the model: training, development, and test data. in the preprocessing step, we check for blank rows and columns and remove such rows and columns from the dataset. we also searched for and removed any rows and columns that lacked text or labels. finally, we converted all the labels into integers for processing. we passed the preprocessed data through two different paths to extract two different types of features. we used the tf-idf and bert algorithms for feature extraction. tf-idf captures the importance of words and tokens based on their frequency in the corpus. the tf-idf feature extraction module was followed by an pca module for dimensionality reduction. this algorithm uses 42 principal components for sampling features from a relatively lower dimensional space. bert extracts contextual information from the input sequence and transforms the input text into a vector representation. here, we used muril @xcite for generating the embeddings. muril is trained over translated and transliterated data in addition to the text in the original script. this motivated us to use muril embeddings for this task. we combine these two dense and lowerdimensional representations in a feature fusion step. we concatenated the contextual embeddings and dimensionality-reduced tf-idf embeddings in this step. we performed a 5-fold cross-validation to ensure the model's robustness. later, we trained the model using the training data and evaluated its performance using the development data. finally, we used the model to predict the labels of the test data. ## experimental results we evaluated our hybrid model on eight indian languages using average f1-score and accuracy as the primary metrics. the results indicate that the model performs consistently well across most languages, especially in high-resource conditions. the best performance was observed for hindi, followed by english and malayalam. these results suggest that the model effectively captures linguistic features and contextual nuances in these languages. the performance of the proposed system on the test data is shown in figure 3 . from the results, it is evident that the model overfits on all languages except gujarati, kannada, and telugu, showing strong generalization capabilities, even in moderately resourced or code-mixed environments. marathi had the lowest f1-score among all languages, indicating possible challenges such as limited data, quality issues, or class imbalance. kannada showed slightly lower results compared to others, but still maintained a reasonable performance. overall, the combination of tf-idf and bert embeddings has proven effective for multilingual hate speech classification. these results highlight the need for further work on handling dataset imbalance and enhancing model performance for lowresource languages. the hyperparameters used for building the model are listed in ## limitations the fusion of tf-idf and bert features can lead to feature redundancy or overfitting, as tf-idf captures frequency-based semantics while bert captures context, potentially introducing conflicting signals. we used the general bert model for feature extraction, which may cause underperformance on low-resource or morphologically rich languages, especially if no multilingual fine-tuning is done.