Build Custom Computer Vision Applications
Tags: computervision, ai, machinelearning, datascience
Published: 2024-02-08 19:06:04

_Author: [Jacob Marks](https://www.linkedin.com/in/jacob-marks/) (Machine Learning Engineer at [Voxel51](https://voxel51.com/))_

## Embed Custom UIs Directly into Data-Centric Workflows

Welcome to week three of Ten Weeks of Plugins. During these ten weeks, we will be building a FiftyOne plugin (or multiple!) each week and sharing the lessons learned!

If you’re new to them, FiftyOne Plugins provide a flexible mechanism for anyone to extend the functionality of their FiftyOne App. You may find the following resources helpful:

- [FiftyOne Plugins Repo](https://github.com/voxel51/fiftyone-plugins)
- [FiftyOne Plugin Docs](https://docs.voxel51.com/plugins/index.html#downloading-plugins)
- [Plugins Channel](https://fiftyone-users.slack.com/archives/plugins) in the [FiftyOne Community Slack](https://slack.voxel51.com/)

Here’s what we’ve built so far:

- Week 1: [AI Art Gallery](https://github.com/jacobmarks/ai-art-gallery) & [Twilio Automation](https://github.com/jacobmarks/twilio-automation-plugin)
- Week 2: [Visual Question Answering](https://github.com/jacobmarks/vqa-plugin)

Ok, let’s dive into this week’s FiftyOne Plugin!

## Custom YouTube Player Panel 📺🔧🔘


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/e86wxd99lennamue02uj.gif)

Over the first two weeks of this 10 Weeks of [Plugins journey](https://docs.voxel51.com/api/fiftyone.operators.types.html#module-fiftyone.operators.types), all three of the plugins that I built were pure Python plugins plugins that allow you to create operators, which execute custom code behind the scenes. These plugins enable a plethora of workflows, and as primarily a Python programmer myself, I find the process for creating Python plugins to be quite intuitive, especially after creating a few.

Sometimes, however, it’s nice to be able to create a custom user interface. Take [VoxelGPT](https://github.com/voxel51/voxelgpt) for instance: the chatbot-like interface (plus easy statefulness!) is really only possible when given its own devoted space within the FiftyOne App. In FiftyOne, creating custom interfaces like this is possible via JavaScript Plugins, which give you blank canvases on which to design workflows and experiences.

In FiftyOne, these “canvases” are called Panels, and when you write a JavaScript plugin, you can build your own panels from scratch. Using JavaScript (and React Material UI components) you can essentially design a custom UI for yourself and/or others, which you can pull up at the click of a button. You’ll see what I mean shortly!

For the third week of _10 Weeks of Plugins_, I built a YouTube Player Panel Plugin. For this plugin, I built a relatively simple UI consisting of a YouTube video player and a few selectors to make it interactive. While this plugin itself is not terribly complex, and my UI is not terribly artistic, it showcases the basic elements required to build a panel to your liking. 

The best part: I used ChatGPT to write the JavaScript for me! 

## Plugin Overview & Functionality

The YouTube Player Panel Plugin is a joint Python/JavaScript plugin with a single operator: `open_youtube_player_panel`, which opens a custom JavaScript component named `YouTubePlayerPanel`. You can choose to play a video from a preset list of YouTube videos from the Voxel51 YouTube channel, or you can play a YouTube video from URL.

There are three ways to execute the  `open_youtube_player_panel` operator and open the panel:

1 - Press the YouTube player button in the [Sample Actions Menu](http://samples_grid_actions):


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/cn00mhx2mfvjxn5a1d7c.gif)

2 - Click on the + icon next to the `Samples` tab and select `YouTube Player` from the dropdown menu:


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/85mfrzi97fjpmctc6jt9.gif)

3 - Press " `` ` `` " to pull up your list of operators, and select "`open_youtube_player_panel`":


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/j23b5n7koltcv64kh5xm.gif)

The Python portion of the plugin is exceedingly simple:

- Storing an SVG icon for YouTube in an `assets` folder within the plugin’s directory, I specified this as the icon to use to represent the operator in two ways: first, in the operator’s `config()` method, so that this would be the icon in the operators list, and second, in its `resolve_placement()` method, so that the button in the action menu is also the YouTube icon.

- In the `execute()` method, the operator triggers an `open_panel` action, and the params passed into `trigger()` instruct that the panel to be opened is named `YouTubePlayerPanel`.

The JavaScript portion is a bit more involved. It consists of:

- `YouTubeIcon`: the SVG icon that appears next to the label `YouTube Player` on the tab in the panel.
- `YoutubeEmbed`: an [iframe](https://www.techtarget.com/whatis/definition/IFrame-Inline-Frame#:~:text=An%20inline%20frame%20(iframe)%20is,webpage%20within%20the%20parent%20page.) component (basically an embedded live view of HTML loaded on another page — in this case YouTube). This component takes as an argument an `embedId`, which is the unique identifier for YouTube videos.
- `YouTubePlayerPanel`: the custom panel containing the YouTube player iframe, a list of preset videos, a text field for inputting a different video link, and the logic for updating the display based on values of the associated variables.
- `registerComponent`: this registers the `YouTubePlayerPanel` we defined as a `Panel` with the name “YouTubePlayerPanel”, so that we can reference it (and open it) from our Python operator.

## Installing the Plugin

If you haven’t already done so, install FiftyOne:

```
pip install fiftyone
```

Then you can download this plugin from the command line with:

```
fiftyone plugins download https://github.com/jacobmarks/vqa-plugin
```
Refresh the FiftyOne App, and you should see the YouTube button show up in the actions menu. 

## Lessons Learned

### Icon Rendering

FiftyOne plugins are very flexible when it comes to icon rendering. If you desire, you can have distinct icons for an operator in the action menu, the operator list, and at the top of its panel. Even though this particular plugin only uses one icon, I try to stick to the convention of putting all icons in an `assets` folder, and then referencing the svg file via absolute path within the plugin’s directory. For example: 

```
icon="/assets/youtube.svg"
```

For the icon in the panel itself, I had to create a separate icon component in JavaScript. The paths in the svg were the same, but I had to use different `viewBox` coordinates to make all of the icons line up nicely in their respective contexts.

### Trigger Composes Operators

In the [AI Art Gallery](https://github.com/jacobmarks/ai-art-gallery) and [Twilio Automation](https://github.com/jacobmarks/twilio-automation-plugin) plugins, I had used the `ctx.trigger()` method to perform operations like reloading samples (`ctx.trigger(“reload_samples”)`), and reloading the dataset (`ctx.trigger(“reload_dataset”)`). I was even aware from [VoxelGPT](https://github.com/voxel51/voxelgpt) that you could use `ctx.trigger()` to set the session’s view.

But it wasn’t until this YouTube Player Plugin that it clicked for me that `ctx.trigger()` [allows you to compose operators](https://docs.voxel51.com/plugins/index.html#operator-composition). You can trigger a Python operator from JavaScript, or a JavaScript operator from Python. You can even trigger operators defined in other plugins!

In this plugin the Python operator’s `execute()` method looks like:


```
def execute(self, ctx):
        ctx.trigger(
            "open_panel",
            params=dict(
                name="YouTubePlayerPanel", isActive=True, layout="horizontal"
            ),
        )
```

What this means is that when executed, the `open_youtube_player_panel` operator is triggering another operator, `open_panel`, with the specific input parameters. In this case, the `open_panel` operator is opening the panel named `YouTubePlayerPanel`, which we defined in our `YouTubePlayerPlugin.tsx` file, and is splitting the canvas of the FiftyOne App in two horizontally (as opposed to vertically).

You can do so much with these operator compositions. I’m excited to explore this further in coming weeks!

### No JavaScript?; No Problem! 

My biggest takeaway from building this plugin is that you don’t actually need to know any JavaScript, React, or Typescript to build a “JavaScript” plugin in FiftyOne. The greatest strength of FiftyOne’s JavaScript plugins is their straightforward interface. 

Because FiftyOne plugin components map fairly directly onto React Material UI components, I found that the [FiftyOne Plugin](https://docs.voxel51.com/plugins/index.html) docs plus the [Material UI docs](https://mui.com/material-ui/getting-started/) plus ChatGPT was more than enough to turn my vision into reality. If you’d like to dive deeper into this, let me know and I’ll put together some resources :)

## Conclusion

FiftyOne plugins are incredibly flexible. They give you the blank canvas you need to create tailor-made data-centric machine learning applications. Playing YouTube videos in the FiftyOne App is cool, but the real value is the ability to educate, to showcase workflows, and to create a compelling story — all with data at the forefront!

Stay tuned for the remainder of these ten weeks as we continue to pump out a killer lineup of plugins! You can track our journey in our [ten-weeks-of-plugins repo](https://github.com/jacobmarks/ten-weeks-of-plugins) — and I encourage you to fork the repo and join me on this journey!