Repository: EleutherAI/lm-evaluation-harness
Issue: Improving the lm-eval Interface
Root cause: Unimplemented Feature Gap

# Motivation
The LM Evaluation Harness exemplifies "function over form"; when performing evaluations with the CLI it supplies verbose information detailing every step of the process from dataset acquisition to delivering the final task metrics, however this is often not very user friendly and - for lack of a better term - quite ugly.

Among all users of the harness 99.99% of them will be utilising terminals that support modern features such as glyphs, cursor management, and text styling. As such we are considering the integration of [rich](https://github.com/Textualize/rich) to create a more approachable -- and aesthetically pleasing -- interface for the harness.

# Why Rich?
It supports a variety of features which can take advantage of advanced terminal rendering capabilities, while still maintaining compatibility with "dumb terminals". Such features and potential use cases include:
- Improved table rendering - more readable metric readouts
- Logger handling - collate and colourise logged messages based on their level
- Advanced text styling - introduce word wrap when logging samples to the console, and highlight key information
- Traceback rendering for errors - can provide additional information useful during debugging such as local variables.
- Progress bars - to improve the rendering of building requests, and running loglikelihood or generate until

# Goal
Our end goal is to maintain the lm-eval's form without sacrificing any of the function. We don't want to remove any of the feedback or functionality provided by the current interface, but aim to make it more readable.

This thread will server as a place for suggestions, ideas and feedback, to track the progress of this endeavour, and to collate relevant issues and pull requests.

So far I've done the following:

## Rich Logger Handling
- Enabled by default in the CLI only
- Automatically injects the RichHandler into the root logging object
- Maintains mostly the same format (same datetime format, but file path is on the right to enable alignment)
<img width="1440" height="96" alt="Image" src="https://github.com/user-attachments/assets/52025974-21ce-42b6-90d7-dbe30779cbc9" />

## Rich Table Markdown Rendering
- Enabled by default in the CLI only
- Will have a switch added to the CLI to revert back to raw markdown printing for easy copy/paste
<img width="1440" height="123" alt="Image" src="https://github.com/user-attachments/assets/97170aed-a520-4866-a88c-8d176da80239" />

## Rich Traceback Handling
- Enabled by default in the CLI only
- When verbosity is INFO or lower will print the traceback:
<img width="1440" height="195" alt="Image" src="https://github.com/user-attachments/assets/98d9ada1-fb78-4151-a5cb-5a16e7c56228" />


- When verbosity is DEBUG will print the traceback with locals:
<img width="1440" height="313" alt="Image" src="https://github.com/user-attachments/assets/9fae706d-72b7-44d8-b039-fa5b9623a043" />

## Eval Argument Preview
- Improved visual feedback of arguments
- Auto resizes columns fields to fit all arguments
<img width="1440" height="120" alt="Image" src="https://github.com/user-attachments/assets/635f219a-882b-4518-bf32-89970a387f73" />
<img width="1440" height="231" alt="Image" src="https://github.com/user-attachments/assets/cbbfc1ee-5330-4cf4-9cbe-598c96606517" />

## `--raw_markdown` CLI Switch
- When passed in the CLI will revert task and metric tables to printing raw markdown for easy copy-paste support.
<img width="1440" height="159" alt="Image" src="https://github.com/user-attachments/assets/c9540fce-e732-4d6c-a910-3902580c0ab9" />