This is a story about renice

About a year ago I was having issues getting real-time audio to work and be acceptably in sync with real-time screencast streaming. I was running applications headless on Debian and capturing the output using built-in protocol commands for screencasting, but the audio output of these apps was not covered by the protocol and just being sent to the system sink.

So I built a little pulseaudio-based NodeJS server that used parec to listen to a loopback device to capture the audio output from these applications.

Initially I just streamed audio over an audio/mpeg mp3 endpoint with Connection: keep-alive and an HTML audio element on the client end. This caused issues tho because the streaming was habitually getting out of sync with the video.

Then I decided to chop up the audio into little packets on the serve, assemble it again on the client, and play it with the Web Audio API. This was much better!

Breaking the content into tiny chunks allowed the bandwidth capacity of the channel to be more efficiently utilized. Latency and jitter were greatly reduced, but it was still not good enough. We would get "drift" - a gradual increase in latency over time.

Thinking about it, it seemed like backpressure was the cause. Basically we were pumping as many little packets through the channel as we could produce, when we produced them, with scant care as to whether the receiver could accommodate those. So even tho chunking had improved latency, latency still increased over time, due to occasional congestion in the network, or at the receiver.

Things really hit a turn, and went next level when we introduced the concept of an "ACK". Basically flow signalling, that acknowledged receipt of a packet (or bundle of packets), and gave permission for the sender to transmit another 1 (or n) subsequent packets.

This nearly solved the latency issues. Gone was all drift, at the cost of the occasional packet loss. But as the audio packets were so infinitesimally small from a perceptual standpoint, there was not much impact on perceived audio quality. But by increasing sync between audio and video, perceived quality of service actually increased far more than was impacted by occasional packet loss.

However there was still an approximately 1 or 2 second delay that I could not remove. Try as I might I just couldn't figure out how to decrease this delay. I played with pulseaudio monitors and devices and parameters to no avail. It was really one of those "give up" technical challenges that doesn't budge to any solution.

Things continued like that for a few months, and then, one day, quite unassuming, the idea occured to me that maybe I should consider renicing some of the processes in the chain. I reniced the NodeJS server to a near real time priority. No change. I reniced the pulseaudio process and parec, and BOOM suddenly, the consistent 1 or 2 second lag (that had sometimes blown out randomly to much longer) was gone. Evaporated. Never to return.

I was ecstatic. It was hard to contain my excitement and I didn't try. It was one of those epic bug fixes that's crucial but part of a long and arduous process of trial and error.

And the solution turns out to be simple. Elegant. And OS-based, not disparaging at all, incredible that you can get that lever.

So this is not a lesson, it's a humble ode to the power and majesty of renice.