Repository: stanford-crfm/helm
Issue: LangChain integration
Root cause: Unimplemented Feature Gap

Hi! I was hoping to figure out how to evaluate LangChain's chains (https://github.com/hwchase17/langchain) using Helm

The chain interface just takes in and returns arbitrary key/value pairs, so should hopefully be possible

I was thinking it would be best accomplished by some type of interface that wrapped a chain into the format HELM expects.

Most of the examples I see here pass in URIs/references to models - that won't work amazingly well for chains. Is there any guidance on basically evaluating a user defined model/function?

Thanks for the suggestion! LangChain looks interesting. But you're right that it would be tricky to integrate this with Helm.

I presume that the kind of integration we want is to use Helm to evaluate chains or agents. In chains and agents, there can be multiple stages of making requests and receiving responses from the LLM, where each stage can make use of responses of previous stages when generating requests. In contrast, in Helm, there can only be a single stage of requests. Helm has adapters, which turns a benchmark input into a prompt; you can think of as similar to a one-stage LangChain chain.

If you just want to evaluate a specific chain or agent, you could define a Helm model that wraps around that chain or agent, essentially treating that chain or agent itself as a "higher-order LLM black box". That has some some drawbacks. Helm will only be aware of the input and output of that black box, and not of the actual raw requests and responses to the wrapped LLM. Also, it will be difficult to expose configurable parameters of the chain or agent to end users.

that makes sense. still, to start it would be interesting to see what results look like when treating that chain/agent as a "higher-order LLM black box". are there any instructions for doing that/how would one go about that?

Unfortunately it isn't particularly easy to do that yet, and also we don't have good docs. I have #1219 to improve the workflow and docs. I can update you when that's ready (probably in the next few weeks).

In the meantime, you can follow the example of the `simple/model1` model. You'll need to:

- Git clone the repository and set up your development environment following [these instructions](https://crfm-helm.readthedocs.io/en/latest/developer_setup/)
- Define a new `Model` [here](https://github.com/stanford-crfm/helm/blob/8f69e30f5a290911b868b118d149150b76615b83/src/helm/proxy/models.py#L561)
- Define a new client like `SimpleClient` [here](https://github.com/stanford-crfm/helm/blob/106113fc7fbd63e50404965ffdec1ed52a45586c/src/helm/proxy/clients/simple_client.py#L15), which will contain the model chain logic. ("Client" is a misnomer because inference can happen locally.)
- Add it to `AutoClient` [here](https://github.com/stanford-crfm/helm/blob/106113fc7fbd63e50404965ffdec1ed52a45586c/src/helm/proxy/clients/auto_client.py#L88-L89) and [here](https://github.com/stanford-crfm/helm/blob/106113fc7fbd63e50404965ffdec1ed52a45586c/src/helm/proxy/clients/auto_client.py#L147-L148)
- Add an entry to `schema.yaml` [here](https://github.com/stanford-crfm/helm/blob/106113fc7fbd63e50404965ffdec1ed52a45586c/src/helm/benchmark/static/schema.yaml#L3)

Then you should be able to run the [quick start example](https://crfm-helm.readthedocs.io/en/latest/quick_start/), replacing `huggingface/gpt2` with your model name.

thanks for the instructions! will try it out in next few days

Currently, HELM assumes that each instance gets adapted into a set of requests.  In the future, we'd like to extend this to something more adaptive where each instance gets adapted into a set of requests, and then the responses to those requests can be used to create new requests, etc.

Closing this as infeasible, because HELM doesn't have an way of defining a high-level system to be test in HELM (i.e. a LangChain strategy + a model), and HELM doesn't have a way of capturing a request log from such as system.