Cerebrium + Daily: Simplifying deployments for your AI-powered voice and video apps
Tags: ai, python, webrtc, news
Published: 2023-10-03 17:29:09

_By [Varun Singh](https://www.daily.co/blog/author/varun/)_

_Daily’s developer platform powers audio and video experiences for millions of people all over the world. Our customers are developers who use our APIs and [client SDKs](https://www.daily.co/products/video-sdk/) to build audio and video features into applications and websites._

_Our [AI Week series](https://www.daily.co/blog/tag/ai-week-2023/) looks at how developers can combine real-time video with AI as they build with our platform. Today we're announcing a partnership with [Cerebrium](https://www.cerebrium.ai/), a serverless infrastructure platform for training, deploying, and monitoring machine learning models. You can now run [_daily-python_](https://docs.daily.co/guides/products/ai-toolkit) seamlessly as part of a [Cerebrium application](https://docs.cerebrium.ai/introduction). You can read more about the [Daily Python SDK](https://www.daily.co/blog/introducing-daily-python-an-sdk-for-ai-powered-interactive-video-and-audio/) here._

_Learn more about the topics we’re covering in this AI Week series, and how we think about the potential of combining WebRTC, video and audio, and AI, in our kickoff post. Feel free to click over and [read that intro](https://www.daily.co/blog/ai-week-at-daily-2023-edition/) before (or after) reading this post._

---


As part of our ongoing AI Week series, we are thrilled to unveil our latest partnership with [Cerebrium](https://www.cerebrium.ai/) – a serverless deployment option for the `daily-python` SDK. Cerebrium is making it easy to run `daily-python` alongside hosted AI models in the same container. This development completely eliminates the need of managing underlying infrastructure, giving engineers easy access to hosted ML capabilities along with voice and video streams. 

## Elevate your apps with AI: running `daily-python` on  serverless infrastructure
Bringing AI models into the world of voice and video applications at scale can feel daunting, particularly when these AI models require intensive computational resources. Tasks such as object tracking, object segmentation, video analysis, or speech transcription demand the right mix of I/O, memory, CPU, GPU resources to ensure real-time performance.

With Cerebrium’s serverless deployment, you can sidestep the intricacies of scaling the CPU/GPU and the Kubernetes resources. Plus, you can easily deploy off-the-shelf ML Prebuilt models that are fine-tuned on your data. Cerebrium already offers an extensive library of over 20 [Prebuilt models](https://docs.cerebrium.ai/cerebrium/prebuilt-models/introduction). This off-the-shelf deployment already contains a broad range of [LLMs](https://docs.cerebrium.ai/cerebrium/prebuilt-models/language-models/gpt4all) and generative [voice](https://docs.cerebrium.ai/cerebrium/prebuilt-models/language-models/whisper) and [video](https://docs.cerebrium.ai/cerebrium/prebuilt-models/image-models/stable-diffusion-2), such as: 

- Llama 2
- GPT4All
- OpenAI’s Whisper
- Meta Seamless
- ControlNet
- Stable Diffusion
- Meta’s Segment Anything
- And more


![Cerebrium's Serveless Architecture with daily-python](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/qrsr43q7jrhwazw87a9n.png)

_Cerebrium's Serveless Architecture with `daily-python`_

Here is what `daily-python` within Cerebrium brings to developers:

1. **Real-time media processing**: the ability to send media from the call to an ML model at up to 15 FPS
2. **Quick model inferences**: receive inferences from the model in 100s of milliseconds, since the model is co-located
3. **Instant action**: as soon as the inference is received from the model, be able to send the modified audio and video or a notification message into the call
For those eager to dive into practical examples, the Cerebrium engineering team has thoughtfully prepared an [example repository of daily-demos](https://github.com/CerebriumAI/daily-demos) for you to play with. There are currently three demos:

- [`content-moderation`](https://github.com/CerebriumAI/daily-demos/blob/master/content-moderation/README.md) uses  OpenAI's [CLIP model](https://openai.com/research/clip).
- [`pet-object-detection`](https://github.com/CerebriumAI/daily-demos/blob/master/pet-object-detection/README.md) uses a Ultralytics’ [YOLO v8 model](https://docs.ultralytics.com/) for object detection.
- [`transcription`](https://github.com/CerebriumAI/daily-demos/blob/master/transcription/README.md) uses OpenAI's [Whisper Model](https://openai.com/research/whisper).

My favorite is the [pet detection demo](https://github.com/CerebriumAI/daily-demos/tree/master/pet-object-detection). The server-side does the following: 

- Join the call with [`daily-python`](https://reference-python.daily.co/index.html), subscribing to video streams from each participant
- Detect pets by passing video frames from each participant to the [YOLO v8 model](https://docs.ultralytics.com/) 
- When a pet is detected, the model returns the frame with a bounding box
- `daily-python` sends the frame with the bounding box into the call, allowing everyone to enjoy a dedicated feed of pets!
You can achieve all of this in roughly 15 lines of code and you will not have to fret about latency or scaling!


![daily-python detects a cat in one of the frames, sends it as active speaker](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/csymvbrg9w83yfyiigqv.png)

_`daily-python` detects a cat in one of the frames, sends it as active speaker_

```
# main.py
def predict(item, run_id, logger):
    item = Item(**item)

    # initialize daily-python
    Daily.init()

    # initialize the AI model
    pet_detector = PetDetection()

    #On startup, connect to room with username Pet Detector
    bot_name = "Pet Detector"
    client = pet_detector.client
    client.set_user_name(bot_name)

    # join daily call with room URL
    pet_detector.join(item.room)

    # iterate on video frame from each participant or user
    for participant in client.participants():
        # ignore local frames, these are self-created pet-detected frames
        if participant != "local":
  	        # send participant frames to the pet detector
            client.set_video_renderer(participant, callback = pet_detector.on_video_frame)

# pet_detection.py

# Load the model weights
pet_detection = YOLO("weights.pt")

class PetDetection(EventHandler):

    def on_video_frame(self, participant, frame):
        self.frame_count += 1
        if self.frame_count >= self.frame_cadence:
          self.frame_count = 0
          self.queue.put(frame.buffer)
          worker_thread = threading.Thread(target=self.process_frame, daemon=True)
          worker_thread.start()

    def process_frame(self):
        buffer = self.queue.get()
        IMAGE_WIDTH = 1280
        IMAGE_HEIGHT = 720

        image = Image.frombytes('RGBA', (IMAGE_WIDTH, IMAGE_HEIGHT), buffer)
        image = cv2.cvtColor(np.array(image), cv2.COLOR_RGBA2BGR)

        detections = pet_detection(image)
        if len(detections[0].boxes) > 0:
          plotted_image = plot_bboxes(image, detections[0].boxes, score=False)
          plotted_image = cv2.cvtColor(plotted_image, cv2.COLOR_BGR2RGB)
          is_success, buffer = cv2.imencode(".png", plotted_image)
          image_stream = io.BytesIO(buffer)
          self.camera.write_frame(Image.open(image_stream).tobytes())

        # Indicate that a formerly enqueued task is complete
        self.queue.task_done()
```

Ready to get started with Cerebrium? Here's how:

1. [Sign up for a Daily account](https://dashboard.daily.co/u/signup). You will need to create a Daily room through the [dashboard](https://dashboard.daily.co/rooms/create) or [programmatically](https://docs.daily.co/reference/rest-api/rooms/create-room) using your Daily API key.
2. [Sign up for an Cerebrium account](https://dashboard.cerebrium.ai/), since we will need to get our API keys to deploy this example.
3. [Git Clone the repository](https://github.com/CerebriumAI/daily-demos) and install the necessary packages by running these commands in the terminal
```
pip install --upgrade cerebrium
cerebrium login <private_api_key> 
cerebrium deploy --config-file ./config.yaml petdetection 
```
This already bundles `daily-python`.

- Manually invite the pet bot to the call by using REST API.
```
curl --location --request POST 'https://run.cerebrium.ai/v3/p-xxxx/pet-detection/predict' \
--header 'Authorization: <JWT_TOKEN>' \
--header 'Content-Type: application/json' \
--data '{"room": "Your Daily Room URL"}'
```
This manual step can be later automated by listening to the [`participant-joined` webhook](https://www.daily.co/blog/configure-a-webhook-to-send-notifications-when-someone-joins-your-video-calls/).

- Invite people to join the created Daily room URL, especially those that have pets available at hand!

One of our north stars is developer time to value, in this case, how much time it takes for a developer with a cool application for an ML model to build, deploy, and start field testing. With our announcement today, what used to take developers weeks can now take minutes. You can get started with Cerebrium’s free tier and Daily’s 10,000 free monthly minutes.

Let us know how we can help support you as you build. Reach out to our [developer support](https://daily.co/contact/support) or head over to [peerConnection](https://community.daily.co/), our WebRTC forum. You can always find us online or IRL at one of our regularly-hosted [events](https://www.daily.co/resources/events/). And check back for more this week in our [AI Week series](https://www.daily.co/blog/tag/ai-week-2023/).