Spaces:
Sleeping
github: https://github.com/SujalNeupane9 linkedin: https://www.linkedin.com/in/sujal-neupane-2a9a2b210/
Summarization
This project is a machine learning pipeline for natural language processing tasks. It contains a set of scripts and modules that allow you to train and evaluate various models on your own data.
Description
This repository contains a sample code with aim to demonstrate how to train a model for text summarization. The main focus is to show a basic template on how to create a structure from which we can smoothly deploy the model as well as perform inference on the trained model.
Framework used:
- PyTorch
- Transformers
Project Structure
pipeline
This directory contains the code for the main data pipeline.
training_pipeline.py
: Code for the training pipeline.inference_pipeline.py
: Code for the inference pipeline.
steps
This directory includes various steps involved in the data pipeline.
evaluation.py
: Code for evaluating the model.ingest_data.py
: Code for ingesting data into the pipeline.preprocess.py
: Data preprocessing code.model_train.py
: Model training code.
utils
This directory contains utility functions used throughout the project.utils.py
: General utility functions.
run_pipeline.py
This script is the entry point for running the entire data pipeline.Dockerfile
The Dockerfile for creating a Docker image for this project.requirements.txt
List of Python packages required for running the project. Install them using:
Demo
I have already trained a t5-base model and uploaded it into HuggingFace. The streamlit demo can be accessed from following link. https://summarization-2s9wr7njxcgpeuuraprip5.streamlit.app/
License
This project is licensed under the MIT License - see the LICENSE file for details.