star092304/ielts-writing-task2-transgat

TransGAT for Automated Essay Scoring

This project implements the TransGAT model for Automated Essay Scoring, leveraging a combination of Transformer architectures and Graph Attention Networks (GATs). It is based on the research presented in the referenced paper.

Multi-Dimensional Scoring (IELTS Task 2 Criteria)

Unlike traditional systems that output a single holistic score, this model performs Multi-Dimensional Automated Essay Scoring. It evaluates essays across the 4 official IELTS Writing Task 2 assessment criteria:

• Task Achievement (TA): Evaluates how well the essay addresses all parts of the prompt and presents a clear, well-supported position.
• Coherence and Cohesion (CC): Assesses the logical structure of the essay, paragraphing, and the effective use of cohesive devices (linking words).
• Lexical Resource (LR): Evaluates the range, accuracy, and appropriateness of the vocabulary used.
• Grammatical Range and Accuracy (GRA): Measures the variety of grammatical structures deployed and the correctness of sentence construction.

The TransGAT architecture models these specific criteria by capturing both semantic features (via Transformers) and structural/relational dependencies within the text (via Graph Attention Networks).

Dataset

The model is trained and evaluated on the IELTS Writing Task 2 Evaluation dataset. You can find the dataset on Hugging Face here: chillies/IELTS-writing-task-2-evaluation 👉 View Clean Dataset

Training Results

Below are the training metrics from the two phases of model training:

Phase 1: Initial Training Metrics

Phase 2: TransGAT Metrics

Usage

You can use this model either by cloning the repository and using the high-level TransGATScorer interface, or by loading the model directly using Hugging Face's transformers library.

Option 1: Using the High-Level Interface (Recommended)

The repository includes an inference.py script that handles the end-to-end pipeline, including text tokenization, dependency graph parsing via Stanza, and score denormalization.

In a Python Notebook / Script:

# 1. Download inference.py, modeling_transgat.py, and graph_utils.py into your workspace
# 2. Run the following code:
from inference import TransGATScorer

# Load the scorer (It will automatically download the model weights and initialize Stanza)
scorer = TransGATScorer("star092304/ielts-writing-task2-transgat")

# Predict scores for an essay
essay_text = "In recent years, climate change has become one of the most pressing global issues..."
result = scorer.predict(essay_text)

print(result)
# Output: {'TA': 7.0, 'CC': 6.5, 'LR': 7.0, 'GRA': 6.5, 'OverallBand': 7.0}

Via Command Line Interface (CLI):

python inference.py --model_dir "star092304/ielts-writing-task2-transgat" --essay "Your essay text here"

Option 2: Loading Directly via Transformers

If you want to integrate the core model into your own custom training or evaluation pipeline, you can load it directly using AutoModel.

⚠️ Note: Since TransGAT requires a dependency graph as an additional input, you will need to manually extract node features (graph_x), edge indices (graph_edge_index), and batch indicators (graph_batch) before feeding them into the model.

import torch
from transformers import AutoTokenizer, AutoModel

# Load tokenizer and custom TransGAT model
model_name = "star092304/ielts-writing-task2-transgat"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModel.from_pretrained(model_name, trust_remote_code=True)

model.eval()

# Prepare text inputs
essay = "Your IELTS essay text here..."
inputs = tokenizer(essay, max_length=512, truncation=True, padding="max_length", return_tensors="pt")

# [Required] You must construct the graph inputs (graph_x, graph_edge_index, graph_batch)
# based on your dependency parser (e.g., Stanza) before running the forward pass.
with torch.no_grad():
    outputs = model(
        input_ids=inputs["input_ids"],
        attention_mask=inputs["attention_mask"],
        graph_x=graph_x,                  # Node features aligned with word embeddings
        graph_edge_index=graph_edge_index, # Graph structure edges
        graph_batch=graph_batch            # Graph batch mapping
    )
    
    # Raw normalized predictions
    logits = outputs["logits"] 

Dependencies

To run the inference smoothly, please ensure you have the following libraries installed:

pip install torch transformers stanza numpy

Reference Paper

This implementation is inspired by and based on the following research paper:

TransGAT: Transformer-Based Graph Neural Networks for Multi-Dimensional Automated Essay Scoring Hind Aljuaida, Areej Alhothalia, Ohoud Al-Zamzamia, Hussein Assalahib

View paper