Kani: A Lightweight and Highly Hackable Framework for Building Language Model Applications (2023)

Abstract: Language model applications are becoming increasingly popular and complex, often including features like tool usage and retrieval augmentation. However, existing frameworks for such applications are often opinionated, deciding for developers how their prompts ought to be formatted and imposing limitations on customizability and reproducibility. To solve this we present Kani: a lightweight, flexible, and model-agnostic open-source framework for building language model applications. Kani helps developers implement a variety of complex features by supporting the core building blocks of chat interaction: model interfacing, chat management, and robust function calling. All Kani core functions are easily overridable and well documented to empower developers to customize functionality for their own needs. Kani thus serves as a useful tool for researchers, hobbyists, and industry professionals alike to accelerate their development while retaining interoperability and fine-grained control.

## introduction we introduce kani, an open-source 1 framework for building language model (lm) applications. kani takes care of the basics of chat interactionquerying models, managing chat history, and calling external functions-allowing developers to write robust application code that is interoperable across any underlying language model. from this minimal base, developers can easily override the core features to implement more complex functionality like retrieval, web hosting, dynamic model routing, and tool usage tracking. unlike existing frameworks, kani is lightweight and highly hackable, allowing developers to control their prompts, customize their models, and handle what's the weather like in tokyo? ## getting started with kani let's start by discussing the basics of installing and querying language models with kani. to start, kani requires python 3.10+ and is installed via pip. this command will install the core kani dependencies. in order to use our pre-built engine classes for huggingface or openai ( ## framework comparison in this section, we compare kani with simpleaichat @xcite and langchain @xcite to highlight kani's strengths (see lightweight. kani is minimalist in both functionality and footprint: we implement essential features with fewer dependencies and less library-specific tooling while accomplishing more (see paired with our detailed documentation, kani's lean and efficient core of features allows developers to start easily and grow rapidly. ## developing applications with kani now that we understand kani's place in the broader ecosystem of tools, we will dive deeper into exactly how to develop lm applications with kani. ## function calling until this point, kani objects had no abilities beyond text generation. function calling (or "tool usage") makes kani objects even more powerful as intelligent assistants. ## customization kani is built on the philosophy that the developer should be in control of every aspect of an application. to accomplish this, kani allows you to override and customize virtually all default behaviors of the library code. in this section we will briefly go over some common customizations developers may want to make. ## advanced usage in this section, we'll look at some more advanced examples. for each of these use cases, we provide the full implementation in the github repository. foot_6 ## conclusion in this paper we presented kani, a lightweight and highly customizable framework for building chat applications. at its core, kani lets developers use the same application code across all language model backends and robustly implements convenient quality-of-life features like chat history management, function validation, and error handling. we believe that the design of our tools is as important as the tools themselves. well-designed tools impose far less friction when using them, freeing up developers' hands from fighting bugs and racking up tech debt. this is especially important now that the llm landscape is so turbulent with new and improved models being released more often than ever. kani offloads the burden of tedious language model management without locking developers into onerous default paradigms, giving developers back control over their applicationshopefully making the landscape a bit less turbulent.