When was the last time you learn something from your LLM logs ? Here is the solution : phospho
Tags: llm, analytics, opensource, ai
Published: 2024-03-06 03:27:37

_TL;DR: [phospho](https://github.com/phospho-app/phospho) is an open-source text analytics for LLM Apps. It helps companies turn their LLM prototype into a product with testing, evaluation, monitoring and guardrails at the semantic level._

Building LLM apps has never been easier. There are TONS of tools. Yet, companies that ship to production are **scarce**. And lots of AI tools that have made it to production have either **HIGH churn rate** or, **LOW usage rate**. Why ?

Unfortunately, many AI builders (I was one of them) are trapped at ground zero:

- They don’t know what to improve in their products, because there is so many ways to improve (and many yet to come!)
- To make decisions, they either have **KPIs** irrelevant to their use case or just **gut feeling** from everyone, but their users
- Who are the users, what they do, or what they say, is usually a **big unknown**.

No wonder they feel stuck. Their best chance at improving is either guesswork or reading through thousands of logs & messages. It is like looking for a needle in a haystack. 

There is no secret. Here is what the best companies shipping LLM products do that others don’t:

- They release often, and fast… because they have a **clear set of custom metrics** that act as a simple “green light/red light”
- They improve on precise product issues… because they **understand in great detail** who use their products and why
- They act on feedback quickly… because they **listen to it every day** in their Slack channels or via mail and **get alerted** when something is going wrong

🧪 This is the purpose of phospho. phospho is an **[open-source text analytics tool for LLM apps](https://github.com/phospho-app/phospho)**.

It gathers all the tools that enable your team to go from prototype to product at record pace: testing, evaluation, monitoring at the semantic level. Let’s deep dive:

### Build and Test
![phospho test](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/xk0ig4hru988fzofk811.png)

- Define your own textual event detection pipeline
- Set up webhooks and enforce guardrails
- Assess the quality before releasing with personalized evals, continuously A/B test

### Understand and Analyze
![phospho dashboard](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/mqcqwp2cpsvl3ytnla5k.png)
- Detect usage patterns, categorize interactions by type, topics, intents, and more
- Evaluate app response quality
- Run tests at scale and in real-time

### Improve and Take Action
![phospho events webhook](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/hgkyd59kga6rsi9j8ik2.png)
- Trigger workflows, escalations, and alerts based on detected events or evaluations
- Dive deeper into the data; get consolidated reports through the platform or via the API
- Break down the analysis by users or sessions

### Integrations

**[Python](https://docs.phospho.ai/integrations/python/logging)** and **[Javascript](https://docs.phospho.ai/integrations/javascript/logging)** SDKs to easily integrate in your LLM stack

Phospho can be **[self hosted](https://github.com/phospho-app/phospho)** or used in a **[managed cloud version](https://phospho.ai/)**

### How to use this text analytics tool ?

✨ The repo is open-source on **[Github](https://github.com/phospho-app/phospho)**. Join the **[Discord](https://discord.com/invite/MXqBJ9pBsx) here**

⚙️ If you run an LLM app, try the platform for **[free](https://phospho.ai/)**.