Semantic Tests for SemanticKernel Plugins using skUnit
Tags: openai, semantickernel, csharp, dotnet
Published: 2024-01-04 14:20:28

## Exploring SemanticKernel
This week, I had the chance to explore the [SemanticKernel](https://github.com/microsoft/semantic-kernel) code base, particularly the core plugins. SemanticKernel comes equipped with these **built-in plugins**:
 - **ConversationSummaryPlugin**
 - FilePlugin
 - HttpPlugin
 - MathPlugin
 - TextPlugin
 - TimePlugin
 - WaitPlugin

When I looked at the `Plugins.UnitTests` project, I noticed that all the unit tests are passing. But there's something interesting:
> Each plugin has a corresponding test file, **except for ConversationSummaryPlugin**. 

You might wonder why!?

Here's the thing. All the other plugins have outputs that can be tested because they're deterministic. But `ConversationSummaryPlugin` is a different story.

For instance, it has a function called `SummarizeConversation` that does exactly what it says - it summarizes a conversation. **But how do you test something like that?** You need to check the meaning of the output, not just if the strings are identical.

Let's consider this test case:

```md
USER: Is Eiffel tall?
AGENT: Yes it is
USER: What about Everest mountain?
AGENT: Yes it is tall too
USER: What about a mouse?
AGENT: No it is not tall.
```

If you call `SummarizeConversation` with this input, you should get something like:

> **Expected output:** The conversation is about the heights of different things. Both the Eiffel Tower and Mount Everest are considered tall, while a mouse is not.


But how do you write a test for that? You need to use semantic assertions, something like:

```csharp
SemanticAssert.HasCondition(
   output, 
   "It mentions that both the Eiffel Tower and Mount Everest are tall.")
```

While you can do this now with the [SemanticValidation](https://github.com/mehrandvd/SemanticValidation) library, I'm going to introduce an even simpler way in this post: using the [skUnit](https://github.com/mehrandvd/skunit) library for semantic unit testing. Sounds exciting, right?

## Let's Dive into Testing with skUnit
With [skUnit](https://github.com/mehrandvd/skunit), you can whip up scenarios in markdown files. Here's an example:

```markdown
# SCENARIO Height Discussion

## PARAMETER input
USER: Is Eiffel tall?
AGENT: Yes it is
USER: What about Everest mountain?
AGENT: Yes it is tall too
USER: What about a mouse?
AGENT: No it is not tall.


## ANSWER
The conversation revolves around the heights of different things. Both the Eiffel Tower and Mount Everest get the tall vote, while a mouse doesn't.

## CHECK SemanticCondition
It mentions that both the Eiffel Tower and Mount Everest are tall.

## CHECK SemanticCondition
It mentions that a mouse isn't tall.
```

As you can see, in a scenario, you can set the parameters and the expected answer. Then, you can specify the semantic conditions that the output should meet. The best part? **skUnit can run this test for you automatically**, and you can see it acing the test. How cool is that?


![Test result](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/l13oazx2b27ejnp97hyv.png)

What’s great is that these scenarios are valid .md files. This means they’re not just for the tech-savvy among us - anyone can read and understand them! Isn’t that neat?


![Markdown](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/v5uw1ykjn1owxad5sg3d.png)



## Finally

I enjoy writing semantic tests for SemanticKernel plugins, and I have created a repository to share some of them: https://github.com/mehrandvd/semantic-kernel-skunit-tests

You can see an example of a test scenario for the SummarizeConversationPlugin [here](https://github.com/mehrandvd/semantic-kernel-skunit-tests/blob/main/SemanticKernel.skUnit.Tests/SemanticKernel.Plugins.Core.Tests/SummarizeConversationPluginTests/Scenarios/EiffelChat.md).