Mistral-7B Academic Metadata Extractor

A fine-tuned Mistral-7B-Instruct-v0.3 model for structured metadata extraction from academic documents.

The model extracts metadata from academic text and produces structured JSON output.

Extracted Metadata

The model extracts the following fields:

  • Title
  • Keywords
  • Authors
  • Affiliations
  • Publisher
  • Journal
  • Publication Year

Performance

The fine-tuned model achieved an overall field-level accuracy of 91.8% on the test set.

Metadata Field Accuracy
Title 98%
Keywords 85%
Authors 96%
Affiliations 93%
Publisher 90%
Journal 88%
Year 87%
Overall 91.8%

Training Dataset

A dataset of 10,000 academic PDF documents was prepared for the project.

The final dataset was divided into:

  • Training: 9,000 documents
  • Validation: 900 documents
  • Testing: 100 documents

Approximately 3,000 documents were manually reviewed to complete or correct missing metadata fields.

Training Configuration

Parameter Value
Base Model Mistral-7B-Instruct-v0.3
Fine-Tuning Method LoRA
Epochs 3 (early stopping at 1.97)
Batch Size 2
Learning Rate 2e-5
Maximum Sequence Length 1024 tokens
GPU NVIDIA A100 80 GB
CPUs per Task 6
Framework Unsloth + Hugging Face

Output Format

The model is designed to return metadata in JSON format:

{
  "title": "",
  "keywords": [],
  "authors": [
    {
      "name": "",
      "affiliations": [""]
    }
  ],
  "publisher": "",
  "year": "",
  "journal": ""
}

Research

This model was developed as part of the research:

Developing AI Agents That Extract Metadata Using The RAG Method With Fine-Tuned Large Language Models

Authors: Tamam Alhasan and SavaÅŸ Takan
Affiliation: Artificial Intelligence and Data Engineering Department, Ankara University

The research compared RAG-based metadata extraction with supervised fine-tuning. Mistral-7B-Instruct-v0.3 achieved 74.43% overall accuracy in the RAG setup, while the fine-tuned model improved the overall accuracy to 91.8%.

Intended Use

This model is intended for extracting structured bibliographic metadata from academic and scientific documents.

Potential applications include:

  • Academic repositories
  • Digital libraries
  • Research databases
  • Automated document indexing
  • Scientific knowledge management

Limitations

The model was trained and evaluated primarily on English-language academic documents. Performance may vary for documents with unusual layouts, incomplete metadata, OCR errors, or domains significantly different from the training data.

Authors

Tamam Alhasan
Artificial Intelligence and Data Engineering
Ankara University, Türkiye

Downloads last month
-
Safetensors
Model size
7B params
Tensor type
F16
·
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for temmam/mistral-7b-academic-metadata-extractor

Adapter
(853)
this model