Repository: UKGovernmentBEIS/inspect_ai
Issue: W&B Plugin

Hi 👋,
I'm sorry if this isn't the right place to reach out.
I saw @jjallaire [mention](https://www.youtube.com/watch?v=_UY49Q_qFhs) that you're going to add a Weights & Biases plugin and I'm interested in helping / connecting someone from W&B to help there. Is there someone I could talk with about this?

Yes! Once this lifecycle hooks PR is merged it will be easy to create an external Python package that implements a W&B plugin: https://github.com/UKGovernmentBEIS/inspect_ai/tree/craig/lifecycle-hooks.

Should be merged within a week or so (cc @craigwalton-dsit). In the meantime I'll direct people to chime in on this thread w/r/t to the features/requirements they'd like to see addressed.



hey @scottire I have a fairly minimal wandb integration that i'm hoping I can replace. The main things it does is:

1. Ensures inspect is outputting logs in a way that doesn't blow up the wandb log view `display=plain` or `display=log`.
2. Gives the wandb run the same ID as the inspect `eval_set` so we can consistently address resources in scripts or the terminal when looking at historical runs
3. Sets the wandb status based on whether the inspect run was successful
4. creates wandb tags that give a quick overview of what we ran e.g. which eval or agent
5. sets wandb config to tell you where the resulting log files were stored e.g. s3 path

what i wish it did:
1. references eval files directly, not the path they live in (it would need to know the eval file name that has the generated task id
2. log dataframes and plots in wandb when the run ends
3. log the same type of stuff when the run is in progress but as samples finish

hey @alex-remedios-aisi, that sounds sweet. I'm also wondering if W&B Weave is suitable here as it's specifically built for LLM Evals. No harm in trying both and seeing which suits your needs better. 

Here's a snippet of logging evals with Weave:

```python
import weave
from openai import OpenAI
from weave import EvaluationLogger 

weave.init('my-project')

# Initialize EvaluationLogger BEFORE calling the model to ensure token tracking
eval_logger = EvaluationLogger(
    model="my_model",
    dataset="my_dataset"
)

# Example input data (this can be any data structure you want)
eval_samples = [
    {'inputs': {'a': 1, 'b': 2}, 'expected': 3},
    {'inputs': {'a': 2, 'b': 3}, 'expected': 5},
    {'inputs': {'a': 3, 'b': 4}, 'expected': 7},
]

# Example model logic using OpenAI
@weave.op
def user_model(a: int, b: int) -> int:
    oai = OpenAI()
    response = oai.chat.completions.create(
        messages=[{"role": "user", "content": f"What is {a}+{b}?"}],
        model="gpt-4o-mini"
    )
    # Use the response in some way (here we just return a + b for simplicity)
    return a + b

# Iterate through examples, predict, and log
for sample in eval_samples:
    inputs = sample["inputs"]
    model_output = user_model(**inputs) # Pass inputs as kwargs

    # Log the prediction input and output
    pred_logger = eval_logger.log_prediction(
        inputs=inputs,
        output=model_output
    )

    # Calculate and log a score for this prediction
    expected = sample["expected"]
    correctness_score = model_output == expected
    pred_logger.log_score(
        scorer="correctness", # Simple string name for the scorer
        score=correctness_score
    )

    # Finish logging for this specific prediction
    pred_logger.finish()

# Log a final summary for the entire evaluation.
# Weave auto-aggregates the 'correctness' scores logged above.
summary_stats = {"subjective_overall_score": 0.8}
eval_logger.log_summary(summary_stats)

print("Evaluation logging complete. View results in the Weave UI.")
```

If it would be helpful to have a call with us to collaborate on this, that suits me 

probs worth going through our account mgr for this - we don't have weave atm and are more focussed on tracking metadata (the inspect toolchain has quite a bit of overlap with weave so we're not missing it too badly atm)

ok cool

Re these:
> what i wish it did:
> references eval files directly, not the path they live in (it would need to know the eval file name that has the generated task id
log dataframes and plots in wandb when the run ends
log the same type of stuff when the run is in progress but as samples finish

Any help needed there? 


yep, we need to define hooks that will support these well then implement a plugin that uses these hooks!

Hooks extension point is now merged to main: https://inspect.aisi.org.uk/extensions.html#hooks

myself and a couple of other collaborators are developing an integration of Inspect with Weave using the hooks interface as part of our [MARS](https://www.cambridgeaisafety.org/mars) project. our current hypothesis is that Weave nicely augments Inspect with easier shareability and some useful visualisation/analysis tooling for low extra effort, but we are actively trying to gather all the possible use cases so I'm keen to hear if anyone has other ideas about the possible synergies between Inspect and Weave

@alex-remedios-aisi we are also considering building in an integration with the wandb models product, especially if it seems like that would be complimentary to an integration between Inspect and Weave. I think it would benefit us a lot if we could have 30 mins of your time to chat through your existing integration and the use cases it fills - if that's something you'd be open to, please reach out to me directly on the Inspect Community Slack

the project can be found [here](https://github.com/DanielPolatajko/inspect_weave)

I am going to close this given that work on the plugin is well underay!