Repository: confident-ai/deepeval
Issue: DeepEval callbacks for LM text-generation quality evaluation during training/fine-tuning
Root cause: Unimplemented Feature Gap

## Problem
During the training or fine-tuning of a language model (LM) using Hugging Face's `Trainer` class or any other framework, there is a need for a flexible mechanism to evaluate text-generation quality through various metrics provided by DeepEval. This can be achieved by introducing a simple callback.

## Solution
The proposed code structure for incorporating this functionality is outlined below:

```python
from deepeval.callbacks.huggingface import DeepEvalCallback
from deepeval.test_case import LLMTestCase
from deepeval.dataset import EvaluationDataset
from deepeval.metrics import HallucinationMetric, AnswerRelevancyMetric
from transformers import Trainer, TrainerArguments

# Model setup from transformers
model = ...
tokenizer = ...

# Setup evaluation dataset
test1 = LLMTestCase(...)
test2 = LLMTestCase(...)
eval_dataset = EvaluationDataset(test_cases=[test1, test2])

# Setup metrics
metrics = [
    HallucinationMetric(...),
    AnswerRelevancyMetric(...)
]

# Setup callback
deepeval_callback = DeepEvalCallback(dataset=eval_dataset, metrics=metrics, ...)

# Trainer setup
train_args = TrainerArguments(...)
trainer = Trainer(train_args, callbacks=[deepeval_callback], ...)

# After each epoch, all the metrics are displayed
trainer.train()
```

# Additional Information
1. Discussed and planned the implementation of these callbacks with @Anindyadeep.
2. Intending to implement callbacks initially for Hugging Face's `Trainer` and subsequently extend support for other frameworks like PyTorch Lightning.
3. Additionally, a `DeepEvalHarnessCallback` class for the ongoing harness implementation by @Anindyadeep will be added.

@penguine-ip, please let me know if any further improvement is required. 😁

PR: #368 

This looks great to me :) 

This looks great @Pratyush-exe, couldn't have designed it better myself! Super intuitive to understand and fits in with existing patterns perfectly. Thanks!

I also assume somewhere in the callback handler there's going to be a parameter that indicates how often evaluation is run (ie. after x epoch)?

There's one additional thing to note is, in order to create an `LLMTestCase` you'll need both `input` and `actual_output`. So DeepEval as it is now means you actually can't initialize an evaluation dataset with you existing code since we don't yet have the actual output. 

But don't worry there's nothing to change on your side, I'm going to make changes to support for this edge case (the case where inference/generation is done at evaluation time). 


Thanks @penguine-ip!
1. Yes, I am adding a parameter to set how often you want evaluation to run, it can be done for **x epochs** or **x steps**.
2. As you said, `LLMTestCase` will be a problem here. Are you planning to make the `actual_output` param optional? Please let me know, what are you thinking.

Thanks! 😁

Yup, so actually `LLMTestCase` will not be changed at all, but what will be changed is we will use something called a "Golden" instead: https://docs.confident-ai.com/docs/confident-ai-manage-datasets#what-is-a-golden

This class already exists and you can even find it in the dataset module (although still needs some work). It was first introduced because currently, we allow teams to centralize their evaluation datasets on Confident AI, which works well if you have pre-computed outputs, however this is not always the case. Some teams may choose to store datasets with missing `actual_output`s, and generate them at evaluation time. To support this, a golden is essentially a row in a dataset that is not yet ready for evaluation, but contains all the necessary metadata (eg. the input) to generate an `actual_output`).

The only thing that is missing right now is I need to decide on the best and most straightforward developer experience to convert a golden into a test case. 


Thanks, I will take a look.

Closing this since this is done :) Thanks @Pratyush-exe