Automated Concept Map Extraction from Text (2025)

Abstract: Concept Maps are semantic graph summary representations of relations between concepts in text. They are particularly beneficial for students with difficulty in reading comprehension, such as those with special educational needs and disabilities. Currently, the field of concept map extraction from text is outdated, relying on old baselines, limited datasets, and limited performances with F1 scores below 20%. We propose a novel neuro-symbolic pipeline and a GPT3.5-based method for automated concept map extraction from text evaluated over the WIKI dataset. The pipeline is a robust, modularized, and open-source architecture, the first to use semantic and neural techniques for automatic concept map extraction while also using a preliminary summarization component to reduce processing time and optimize computational resources. Furthermore, we investigate the large language model in zero-shot, one-shot, and decomposed prompting for concept map generation. Our approaches achieve state-of-the-art results in METEOR metrics, with F1 scores of 25.7 and 28.5, respectively, and in ROUGE-2 recall, with respective scores of 24.3 and 24.3. This contribution advances the task of automated concept map extraction from text, opening doors to wider applications such as education and speech-language therapy. The code is openly available.

## introduction concept maps 3.0 @xcite leverage semantic web (sw) technologies to create dynamic concept maps (cms). these summaries of visual graphs represent the semantic relationships between concepts extracted from text, as shown in the concept map extracted in ## related work concept map 3.0 suggests the use of cms enriched by web 3.0 technologies, using sw resources, such as schema.org and wikidata, and following web data principles to make them machine-interpretable and semantic learning resources @xcite . towards this line, we contribute with our neuro-symbolic pipeline for automatic cm extraction that utilizes sw tools. although this task can be broken down into several components, evaluating these individual components is beyond the scope of this task and of our work. we focus solely on complete approaches for cm extraction from text. currently, the literature conventionally portrays automatic cm extraction from text as a multistep approach involving subtasks such as concept and relation extraction and subgraph selection. existing works are twofold: those with a single document as input, namely the cm -document summarization (cm-ds) task @xcite , and those with multiple documents as input, namely the cm -multi document summarization (cm-mds) task. for cm-ds, @xcite laid the foundation not only by extracting relations between concepts from a text file, but also by extrapolating rules about the knowledge at hand. subsequent studies such as @xcite employed unsu- authors task lang. meth.se ir ee re oliveira et al. (2001) s en l ✓ rajaraman and tan (2002) m en l ✓ ✓ cañas and novak (2006) s en ls ✓ kowata et al. (2010) s pr ls zouaq et al. (2011) m en l ✓ ✓ ✓ zubrinic et al. (2012) m cr ls post ✓ ✓ qasim et al. (2013) m en ls ✓ ✓ žubrinić et al. (2015) m cr ls ✓ ✓ de aguiar et al. (2016) s en ls post ✓ ✓ falke (2019) m en,de ls post ✓ ✓ ✓ nugumanova et al. (2021) m en,kk,ru l ✓ ✓ bayrak and dal (2024) m tr ls ✓ ✓ ✓ our pipeline approach m,s en lsn pre ✓ ✓ ✓ pervised methods with deep syntactic parsing for concept selection. these methods primarily used term frequencies to assign a document to the most probable cm among a set of options, enhancing the precision of concept selection. kowata et al. (2010) further focused on extracting cms from portuguese news articles. this work pioneered the use of a comprehensive pipeline approach that included text segmentation, tokenization, part-of-speech tagging, core element candidate recognition, dependency interpretation, and cm construction. subsequently, de aguiar et al. (2016) introduced a sophisticated pipeline approach that integrated grammar rules, co-reference resolution, and concept ranking based on frequency of occurrence. lastly, bayrak and dal (2024) introduced a new heuristic approach to extract cms from turkish texts. for cm-mds, @xcite pioneered the field by using regular expressions and term frequency-based grouping to build a cm-based knowledge base from text documents. they used named entity recognition, extracted noun-verb-noun triples using a pos tagger and handcrafted rules, disambiguated them with word-net @xcite , and clustered them. their approach was integrated into a system and validated through experimental studies. @xcite later defined specific patterns on dependency syntax representations to enhance entity extraction. their work highlighted the usefulness of cm mining in ontology learning. @xcite extended the cm-mds task by introducing a heuristic approach to summarize cms from legal documents written in croatian. this was a significant advance that demonstrated the adaptability of cm-mds techniques to other languages and domain-specific document types. lastly, @xcite @xcite made significant contributions to the field. their model leverages predicate-argument structures and automatic models for german and english, achieving sota performance until now. their pipeline includes five steps: (1) concept and relation extraction, from open information extraction @xcite ; (2) concept mention grouping and labeling with greedy search optimization (3) relation mention grouping, labeling, and selection using lemmatization; (4) importance estimation with a ranking support vector machine; (5) cm construction using integer linear programming @xcite . their english datasets, wiki @xcite and educ @xcite , are the two largest annotated corpora for cm-mds and serve as the main benchmark for this task. wiki was obtained through an automated corpus extension method with automatic pre-processing, crowdsourcing, and expert annotations. it contains 38 groups, each with several documents and focused on a different topic. it is split 50/50 across the training and the test set. each cluster contains on average 15 documents and comes with a cm reference. educ contains 30 document clusters on educational content and was created through crowdsourcing; unlike wiki, the authors had not released this data set for use in this investigation. ## results the lower rouge-2 scores suggest that the pipeline's generated cms do not include the exact words to match the bigrams of the gold standard, leading to a loss in performance @xcite . omitting the ranking module in b resulted in a decrease in rouge-2 scores (f1 of 3.84 instead of 7.61 in a ). the full pipeline a achieves the best f1 rouge-2 performance, showing that the combination of ranking, summarization, and entity extraction is effective for capturing a broader range of n-grams, aligning better with the gold-standard references. across the three pipelines a , b, and c , rouge-2 scores consistently lag behind baselines in the existing literature, particularly in precision, highlighting potential limitations in capturing all pertinent details despite effectively conveying the main points, as indicated by higher meteor scores. this suggests opportunities to improve content coverage and lexical alignment. the higher rouge-2 recall metrics observed in c , which exclude summarization, may indicate that summarization processes introduced new information, such as the generation of words not present in the original text. this could include the use of synonyms or reformulations, which rouge-2 does not ac-count for, as it relies on exact word matching rather than capturing semantic similarities. these observations also raise concerns about the potential loss of critical information during summarization, which should be mitigated in future work. generally, our higher meteor and rouge-2 recall scores indicate improved summarization quality by emphasizing semantic accuracy and readability over exact word overlap, key factors in our educational context. meteor, in particular, captures these aspects more effectively than rouge-2 @xcite @xcite . the lower rouge-2 scores compared to falke et al. may stem from irrelevant or misaligned triples, occasionally resulting in 0.0 scores. moreover, our pipeline significantly enhances efficiency with the summarization component, processing each folder in an average of 13s (wikitrain) and 15s (wiki-test), compared to 40s and 1 minute with the non-summarization pipeline, a 3-4x speedup, as it can be seen in the processing times logs in the github. ## conclusion we propose a neuro symbolic pipeline and a large language model-based method for automated concept map extraction from text evaluated over the wiki dataset. our novelty lies in the architecture that utilizes state-of-the-art tools into a neurosymbolic pipeline with modularized components and its unique application to concept map extraction. our architecture is the first one to combine symbolic, statistical, and neural technique and to have a summarization step at the beginning of the pipeline. key technical contributions are the fine-tuned rebel model and the summarization component, which enhance the originality of the pipeline. moreover, we analyzed end-to-end llmbased approaches, which are the first llm-based end-to-end methods for automated cm extraction. the decomposed prompting method had the best meteor f1 scores and rouge-2 recall, outper-forming the current sota and effectively competing with supervised and unsupervised methods. in future work, our aim is to investigate lexical embeddings and semantic rules to increase the performance and accuracy of cm extraction from text. furthermore, the current metrics used are suitable for text summarization tasks but are not tailored to the cms generation, as they miss critical aspects of cm creation, such as the graph structure and semantically equivalent concepts, suggesting the need for a new metric. we thus plan to explore evaluation metrics and semantically enhanced benchmarks that are more adapted to this task. in particular, we could explore embedding similarity using a pre-trained language model, or we could also adapt taxonomy metrics such as rate and repurposed datasets such as the semeval 2016 task 13 @xcite to evaluate the quality of concept maps. moreover, the pipeline should be evaluated on a broader range of texts, encompassing both general and domain-specific content, to assess its robustness across different contexts and to understand how domain knowledge affects performance. this will involve curating more diverse datasets that enable a thorough evaluation and reveal opportunities for further improvement. additionally, future work should explore post-processing techniques to ensure that key details are preserved in the summarized text, supporting more accurate knowledge representation. a comparative analysis of concept maps generated from both summarized and fulltext versions should be done to examine potential trade-offs and better understand the impact of summarization on the overall quality of the concept maps.