For 1 hour and 42 minutes, the app that tells my stream what I'm playing didn't see a single track. The whole time, its container reported healthy and its overlay page answered every request. I had power-cycled my decks between sets, and the app only looks for decks once, when it starts.
It was the third failure in about two months. Each one had a different cause. One check would have told them apart in seconds: when did the app last detect a track?
What the app does
The companion app runs next to my decks and listens to their network feed. When a new track comes in, it sends an event that fans out to three places: the visuals and camera trigger, the now-playing line in Twitch chat, and play counts in my library. The core of the app is a closed third-party tool I can't patch, so every fix in this post lives around it.
It's not in the audio path. Restarting it takes about 20 seconds and is safe in the middle of a set.
Failure one: healthy and blind
In September the symptoms matched the previous night's exactly, and that night the cause had been further down the chain, so the obvious move was to assume that fix hadn't held and probe downstream. Wrong guess.
The app's process was alive. The container said healthy. The overlay page returned 200. The last track it had detected was 1 hour 42 minutes old.
The app discovers the decks once, at startup, and has no way to reconnect. I had powered the rig off and on between sets while it kept running, so it held on to a session with decks that no longer existed. I already had a start gate that waited for a deck before launching the app, but it only re-armed when the app exited, and a blind app doesn't exit.
The fix that night was a restart. It found both decks right away, and a track went through the whole chain within about half a minute.
The real question is why the health check stayed green. It was set up to pass if the overlay answered or the start gate was waiting. That was on purpose: the rig is off whenever I'm not mixing, and a check that shows red every time the decks are off is one you learn to ignore. Neither half asked whether a track had been seen recently. It measured that the app was up, and the app was up.
Failure two: a log line for a dead connection
In July, the app was posting the now-playing line to chat itself. After a morning restart, the first track landed fine. About half an hour later, it logged three times that it had written a track to Twitch, and nothing showed up in chat. No error anywhere.
So I looked at its connections instead of its log. The app had no outside connections at all, only the links to the decks and its own local listener. Its chat connection had died at some point in that half hour, and from then on it was writing into nothing and logging success. A restart didn't bring it back.
I moved chat posting into a small bot of its own, fed by the same fan-out, and confirmed its first post with a separate read-only listener in the channel, not the bot's log. The app's old log line about writing to Twitch still shows up. It just doesn't mean anything now, and that's a good reason to only trust proof taken at the receiving end.
Failure three: the mixer said hello and shut the door
Late in July the app went into a crash loop. The mixer was on the network and answering discovery, but when the app opened the channel that carries its data, the mixer refused. The refusal stayed through restarts.
That split is the useful part. When a device announces itself and then won't serve, the network has done its job: discovery works, so the box is reachable. The problem is the box: a menu setting or the state the hardware is in. There was no safe way to fix that remotely, so I parked it until I could put hands on the mixer.
Digging into the loop turned up a second cause that was mine: my own listener announced itself on the network with no port, and the app kept trying to connect to it. The container had restarted 541 times. The decks carry everything the app needs, so I put a start gate in front of the app that counts only a beacon from an actual deck, with a real port and deck firmware. The mixer and my own listener don't count. That's the gate from failure one. Tested both ways: 90 seconds of non-deck beacons kept it waiting, and a fake deck beacon started it in under a second.
The parts that were not magic
My own probe lied. One September check piped its output through a trimming command and then read the exit code, which belonged to the trimming step. It reported success for endpoints that were down. If a probe can't fail, it isn't a probe.
I couldn't test it live. The fastest proof is to push a fake track through the chain. While I'm live, that posts a made-up song to chat and swings the camera. I didn't, on either night.
Where I was the bottleneck
I was the one who power-cycled the decks, and the health check that hid it was set up that way to stop false alarms on my rig. It was a reasonable call for gear that's off between sessions, and it asked the wrong question. The mixer needed me standing at the gear, and no remote check could change that.
If you are about to try this
- Check the last-detection time before you probe anything. A long silence behind a healthy status means the listener is blind. A recent detection means look downstream.
- Make the health check assert the job. A recent track while the decks are on is health. A process that answers is only a pulse.
- Prove delivery at the receiver. A sender's log saying it wrote something proves only that it tried.
- Discovery up, data refused: walk over to the device. The network has already done its job.
Build it yourself
What it does: keeps a now-playing chain honest when the thing listening to the decks can go blind without crashing. The pieces:
- A last-detection timestamp the listener updates on every track event, readable from outside.
- A health check that fails when the decks are broadcasting but the last detection is older than your threshold, and passes quietly when the rig is off.
- A start gate that waits for a beacon from an actual deck (real port, deck firmware) and ignores everything else that announces itself.
- A watchdog that restarts the listener when the health check fails, with a cap on restarts per hour.
- Delivery checks at each consumer: a read-only chat listener, the visuals software's own log of what it received.
How it fits: the gate starts the listener, the listener stamps every detection, the health check compares that stamp with whether the decks are on, and the watchdog acts on it. No new gear, only small scripts around the app.
If you've had a monitor stay green through a dead show, I'd like to hear how you caught it.