  We propose NNStreamer, a software system that handles neural networks as
filters of stream pipelines, applying the stream processing paradigm to deep
neural network applications. A new trend with the wide-spread of deep neural
network applications is on-device AI. It is to process neural networks on
mobile devices or edge/IoT devices instead of cloud servers. Emerging privacy
issues, data transmission costs, and operational costs signify the need for
on-device AI, especially if we deploy a massive number of devices. NNStreamer
efficiently handles neural networks with complex data stream pipelines on
devices, significantly improving the overall performance with minimal efforts.
Besides, NNStreamer simplifies implementations and allows reusing off-the-shelf
media filters directly, which reduces developmental costs significantly. We are
already deploying NNStreamer for a wide range of products and platforms,
including the Galaxy series and various consumer electronic devices. The
experimental results suggest a reduction in developmental costs and enhanced
performance of pipeline architectures and NNStreamer. It is an open-source
project incubated by Linux Foundation AI, available to the public and
applicable to various hardware and software platforms.
