We open sourced our AI gateway written in TS
Tags: ai, javascript, typescript, opensource
Published: 2024-01-09 12:26:54

We've been building a robust AI gateway at [Portkey](https://portkey.ai) for the past 10 months.

Over this time, Portkey has been used by developers to test their prompts and measure costs and performance. We have been interface for shuttling 100B tokens and 10M requests daily to 100+ LLMs. 

We've believed that to accelerate Gen AI adoption, companies and AI engineers need strong foundational platforms so going to production is not a nervous affair. The AI gateway in our mind is one of the MOST critical pieces of infrastructure in the new AI stack.

Guess what? Today, we are officially open-sourcing the magic sauce, that got us here. 

## Portkey's Opensource AI Gateway

{% embed https://github.com/Portkey-AI/gateway %}

The AI gateway is an essential component of Portkey's platform. [Portkey's AI Gateway](https://github.com/Portkey-AI/gateway) helps you route requests to multiple LLMs and enables you to build on a unified API. 
 
![Portkey AI Gateway](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/djwzmoe59lfh1v31qmtt.png)

### Why an AI Gateway?

The Generative AI ecosystem has been expanding rapidly (for good!), leading to a lack of resiliency and a disjointed developer experience when building with different components. 

Imagine that you have added a Generative AI feature to your SaaS app, which your users use widely. However, there is a new model available from the Large Language model provider that you want to test before implementing it in production. 

Although you are satisfied with the success of your current AI feature, you still lack insights into the costs and errors from your users.

To address these issues, Portkey's AI Gateway is specifically designed to assist developers.

### Key Features

#### Universal API

AI Gateway offers a universal and unified API that allows interaction with over 100 LLMs, whether privately deployed or public LLM vendors. We take care of request and response transformations automatically.

#### Load Balancing 

Portkey's Load Balancing feature efficiently distributes network traffic across multiple Language Model APIs, preventing any LLM from becoming a performance bottleneck and ensuring high availability and optimal performance of your generative AI apps. 

#### Fallbacks

Several Language Model APIs are available in the market, each with its strengths and specialities. It would be very convenient if the APIs could be easily switched between based on their performance or availability. Portkey's Fallback feature enables one to switch between multiple LLMs.

Just imagine your users are not affected by your LLM Vendor's downtime. 

## Getting Started

The gateway is built in TypeScript and blazingly fast to try out or deploy. 

```javascript
npx @portkey-ai/gateway
```

## Building with AI Gateway

While we understand the promise of universal API, we also acknowledge the learning curve of a new API. 

Most of our SDKs support OpenAI convention and work with Llamaindex and LangChain.

| Language | Supported SDKs |
|---|---|
| Node.js / JS / TS | [Portkey SDK](https://www.npmjs.com/package/portkey-ai) <br> [OpenAI SDK](https://www.npmjs.com/package/openai) <br> [LangchainJS](https://www.npmjs.com/package/langchain) <br> [LlamaIndex.TS](https://www.npmjs.com/package/llamaindex) |
| Python | [Portkey SDK](https://pypi.org/project/portkey-ai/) <br> [OpenAI SDK](https://pypi.org/project/openai/) <br> [Langchain](https://pypi.org/project/langchain/) <br> [LlamaIndex](https://pypi.org/project/llama-index/) |
| Go | [go-openai](https://github.com/sashabaranov/go-openai) |
| Java | [openai-java](https://github.com/TheoKanning/openai-java) |
| Rust | [async-openai](https://docs.rs/async-openai/latest/async_openai/) |
| Ruby | [ruby-openai](https://github.com/alexrudall/ruby-openai) |


## Portkey's LLM Developer Community

Open-sourcing the gateway can lead to more community involvement and innovation, making it even more useful for developers building AI-enabled experiences.

Please join us with hundreds of developers at our [discord](https://discord.gg/MxqgrFqq) to discuss building and contributing with AI. 

We believe that Open-sourcing the gateway can lead to more community involvement and innovation, making it even more useful for developers building AI-enabled experiences.

Please consider giving [Portkey's AI Gateway](https://github.com/Portkey-AI/gateway) a star 🌟. We welcome code and non-code contributions from you - here are some [`good-first-issues`](https://github.com/Portkey-AI/gateway/issues?q=is:open+is:issue+label:%22good+first+issue%22) to start. 

If you have any questions, feedback. Please let us know in the comments below!