Hugging Face's logo

sarkarsudipta
/
bart-base-scihigh

Model card Files
xet
 Community

Fine-Tuned BART-Base for SciHigh 2026 - Task 1

This repository contains a fine-tuned BART-base model developed for SciHigh 2026 - Task 1.

The model is fine-tuned using LoRA (Low-Rank Adaptation) for automatic generation of concise and informative research highlights from scientific abstract.

The trained model checkpoint is provided as:

model_best.pth

Model Details

  • Developer: Sudipta Sarkar
  • Base Model: facebook/bart-base
  • Architecture: BART-base Encoder-Decoder
  • Model Type: Sequence-to-Sequence (Seq2Seq) Transformer
  • Fine-Tuning Method: LoRA
  • LoRA Target Modules: q_proj, v_proj
  • LoRA Rank: 8
  • LoRA Alpha: 16
  • LoRA Dropout: 0.1
  • Language: English
  • Task: Research Highlight Generation
  • Competition: SciHigh 2026 - Task 1
  • Checkpoint: model_best.pth

Task Description

The objective of SciHigh 2026 - Task 1 is to automatically generate a concise research highlight from scientific abstract.

Input

A scientific abstract.

Output

A concise research highlight describing the key information, contribution, or finding from the input text.

The model follows the general sequence:

Scientific Abstract
        |
        v
   BART-base Encoder
        |
        v
   BART-base Decoder
        |
        v
Research Highlight

LoRA is applied to selected attention projection layers, specifically q_proj and v_proj, during fine-tuning.

Base Model

The model is based on the pretrained:

facebook/bart-base

Base model:

https://huggingface.co/facebook/bart-base

Model Checkpoint

The trained checkpoint is available in this repository:

model_best.pth

The checkpoint contains the trained BART model parameters together with the LoRA parameters obtained during fine-tuning.

How to Use

Installation

Install the required packages:

pip install torch transformers peft sentencepiece huggingface_hub
pip install -U "torchao>=0.16.0"

Note: The installed torchao version should be compatible with the installed version of PEFT. Recent versions of PEFT may require torchao >= 0.16.0.

Load the Model and Checkpoint

The following example downloads the checkpoint directly from this Hugging Face repository and reconstructs the LoRA-based BART model.

import torch

from huggingface_hub import hf_hub_download
from transformers import (
    AutoTokenizer,
    AutoModelForSeq2SeqLM
)

from peft import (
    LoraConfig,
    get_peft_model
)


# --------------------------------------------------
# 1. Configuration
# --------------------------------------------------

BASE_MODEL_NAME = "facebook/bart-base"

REPO_ID = "sarkarsudipta/bart-base-scihigh"

CHECKPOINT_FILE = "model_best.pth"


# --------------------------------------------------
# 2. Download checkpoint from Hugging Face
# --------------------------------------------------

CHECKPOINT_PATH = hf_hub_download(
    repo_id=REPO_ID,
    filename=CHECKPOINT_FILE
)

print("Checkpoint:")
print(CHECKPOINT_PATH)


# --------------------------------------------------
# 3. Device
# --------------------------------------------------

device = torch.device(
    "cuda" if torch.cuda.is_available() else "cpu"
)

print("Device:", device)


# --------------------------------------------------
# 4. Load tokenizer
# --------------------------------------------------

tokenizer = AutoTokenizer.from_pretrained(
    BASE_MODEL_NAME
)


# --------------------------------------------------
# 5. Load pretrained BART-base
# --------------------------------------------------

base_model = AutoModelForSeq2SeqLM.from_pretrained(
    BASE_MODEL_NAME
)


# --------------------------------------------------
# 6. Configure LoRA
# --------------------------------------------------

lora_config = LoraConfig(
    r=8,
    lora_alpha=16,
    target_modules=["q_proj", "v_proj"],
    lora_dropout=0.1,
    bias="none",
    task_type="SEQ_2_SEQ_LM"
)


# --------------------------------------------------
# 7. Create PEFT/LoRA model
# --------------------------------------------------

model = get_peft_model(
    base_model,
    lora_config
)


# --------------------------------------------------
# 8. Load trained checkpoint
# --------------------------------------------------

checkpoint = torch.load(
    CHECKPOINT_PATH,
    map_location="cpu"
)

print("Checkpoint type:", type(checkpoint))


# Extract model state dictionary
if (
    isinstance(checkpoint, dict)
    and "model_state_dict" in checkpoint
):
    state_dict = checkpoint["model_state_dict"]
else:
    state_dict = checkpoint


# Load trained parameters
missing_keys, unexpected_keys = model.load_state_dict(
    state_dict,
    strict=False
)


print("Missing keys:", len(missing_keys))
print("Unexpected keys:", len(unexpected_keys))


# --------------------------------------------------
# 9. Move model to device
# --------------------------------------------------

model.to(device)
model.eval()

print("Model loaded successfully!")

Example Inference

The following example demonstrates research highlight generation from a scientific abstract.

# Example scientific abstract

abstract_text = """
       Give your scientific abstract
"""


# --------------------------------------------------
# Tokenize input
# --------------------------------------------------

prompt = abstract_text

inputs = tokenizer(
    prompt,
    return_tensors="pt",
    max_length=512,
    truncation=True
)


# Move tensors to device

inputs = {
    key: value.to(device)
    for key, value in inputs.items()
}


# --------------------------------------------------
# Generate research highlight
# --------------------------------------------------

with torch.no_grad():

    outputs = model.generate(
        **inputs,
        max_length=128,
        num_beams=4,
        early_stopping=True
    )


# --------------------------------------------------
# Decode prediction
# --------------------------------------------------

predicted_highlight = tokenizer.decode(
    outputs[0],
    skip_special_tokens=True
)


print("Predicted Research Highlight:")
print(predicted_highlight)

Training Configuration

The model was fine-tuned using the following configuration:

  • Base Model: facebook/bart-base
  • Architecture: BART Encoder-Decoder
  • Fine-Tuning Method: LoRA
  • LoRA Rank: 8
  • LoRA Alpha: 16
  • LoRA Dropout: 0.1
  • Target Modules: q_proj, v_proj
  • Task: Research Highlight Generation
  • Dataset: SciHigh 2026 - Task 1
  • Language: English

LoRA Fine-Tuning

LoRA (Low-Rank Adaptation) is used to efficiently fine-tune the pretrained BART-base model.

Instead of updating all parameters of the pretrained model, trainable low-rank matrices are introduced into selected attention projection layers.

For this model, LoRA is applied to:

q_proj
v_proj

The resulting checkpoint contains both the pretrained BART parameters and the learned LoRA parameters.

The relevant LoRA parameters include components such as:

lora_A
lora_B

for the selected attention projections.

Model Architecture 

                         Scientific Abstract
                                  |
                                  v
                         +----------------+
                         |   BART Encoder |
                         +----------------+
                                  |
                                  v
                         +----------------+
                         |   BART Decoder |
                         +----------------+
                                  |
                                  v
                       Research Highlight

During fine-tuning, LoRA modules are introduced into selected attention projections:

Attention Layer
       |
       +---- q_proj ---- LoRA
       |
       +---- k_proj
       |
       +---- v_proj ---- LoRA
       |
       +---- out_proj

Intended Use

This model is intended primarily for:

  • SciHigh 2026 - Task 1 evaluation
  • Scientific research highlight generation
  • Scientific text summarization
  • Abstract-to-highlight generation
  • Research-oriented text generation

The model is provided for research and competition purposes.

Limitations

The model was fine-tuned specifically for the SciHigh 2026 Task 1 dataset. Therefore, its performance may vary on scientific text from domains or distributions that differ from the training data.

Generated highlights may contain:

  • factual inaccuracies,
  • omissions,
  • incomplete descriptions, or
  • overly general statements.

Generated text should therefore be evaluated before being used in downstream scientific applications.

Requirements

The recommended environment contains:

pip install torch transformers peft sentencepiece huggingface_hub
pip install -U "torchao>=0.16.0"

The exact PyTorch, Transformers, PEFT, and TorchAO versions should be selected to maintain compatibility with one another.

Repository

Hugging Face Model Repository:

https://huggingface.co/sarkarsudipta/bart-base-scihigh

Base Model:

https://huggingface.co/facebook/bart-base

License

This repository is released under the MIT License.

The underlying facebook/bart-base model is subject to its original license and terms of use.

Acknowledgements

We acknowledge the organizers of SciHigh 2026 and the developers of the Hugging Face Transformers and PEFT libraries.

Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support