README.md

---
title: Commit Rewriting Visualization
sdk: gradio
sdk_version: 4.25.0
app_file: change_visualizer.py
---

Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference

# Description

This project is a main artifact of the "Research on evaluation for AI Commit Message Generation" research.

# Structure (important components)

- ### Configuration: [config.py](config.py)
    - Grazie API JWT token and Hugging Face token must be stored as environment variables.
- ### Visualization app -- a Gradio application that is currently deployed
  at https://huggingface.co/spaces/JetBrains-Research/commit-rewriting-visualization.
    - Shows
        - The "golden" dataset of manually collected samples; the dataset is downloaded on startup
          from https://huggingface.co/datasets/JetBrains-Research/commit-msg-rewriting
        - The entire dataset that includes the synthetic samples; the dataset is downloaded on startup
          from https://huggingface.co/datasets/JetBrains-Research/synthetic-commit-msg-rewriting
        - Some statistics collected for the dataset (and its parts); computed on startup

      _Note: datasets updated => need to restart the app to see the changes._
    - Files
        - [change_visualizer.py](change_visualizer.py)
- ### Data processing pipeline (_note: datasets and files names can be changed in the configuration file_)
    - Run the whole pipeline by running [run_pipeline.py](run_pipeline.py)
        - All intermediate results are stored as files defined in config
    - Intermediate steps (can run them separately by running the corresponding files
      from [generation_steps](generation_steps)). The input is then taken from the previous step's artifact.
    - Generate the synthetic samples
        - Files [generation_steps/synthetic_end_to_start.py](generation_steps/synthetic_end_to_start.py)
          and [generation_steps/synthetic_start_to_end.py](generation_steps/synthetic_start_to_end.py)
        - The first generation step (end to start) downloads the `JetBrains-Research/commit-msg-rewriting`
          and `JetBrains-Research/lca-commit-message-generation` datasets from
          Hugging Face datasets.
    - Compute metrics
        - File [generation_steps/metrics_analysis.py](generation_steps/metrics_analysis.py)
        - Includes the functions for all metrics
        - Downloads `JetBrains-Research/lca-commit-message-generation` Hugging Face dataset.
    - The resulting artifact (dataset with golden and synthetic samples, attached reference messages and computed
      metrics) is saved to the file [output/synthetic.csv](output/synthetic.csv). It should be uploaded
      to https://huggingface.co/datasets/JetBrains-Research/synthetic-commit-msg-rewriting **manually**.
- ### Data analysis
    - [analysis_util.py](analysis_util.py) -- some functions used for data analysis, e.g., correlations computation.
    - [analysis.ipynb](analysis.ipynb) -- compute the correlations, the resulting tables.
    - [chart_processing.ipynb](chart_processing.ipynb) -- Jupyter Notebook that draws the charts that were used in the
      presentation/thesis.
    - [generated_message_length_comparison.ipynb](generated_message_length_comparison.ipynb) -- compare the average 
      length of commit messages generated using the current prompt (one used in the research) and the production prompt 
      (one used to generate the messages that are measured in FUS logs). _Not finished, because could not get a Grazie 
      token; as soon as the token is received, the notebook can be run by following the instructions from the notebook._