The Radio That Lied About Dying

The radio in my new Meshtastic base station was dead for ten days, and the service manager recorded every one of its fourteen restart attempts as a clean, successful exit. Read at face value, that pointed at the hardware, and the first evaluation called the new station "not a candidate today." The real cause was a full node database crashing the radio software, behind a start script that reported each crash as a success. I found it by building the whole station the way I wanted it, with transmit turned off, and watching. That inert build also caught a bug that would have given my node a new identity on every reboot.

A station that looked fine and wasn't

Meshtastic is my playground: cheap LoRa radios that pass text across a mesh with no cell service, and an Arizona community that keeps building it out. In July 2026 I set up a RAK WisMesh Station, a Raspberry Pi with a LoRa board on top, to take over from my older base station: the bots, the dashboard and the radio duties.

I thought it had been running well, including a stretch with its network cable unplugged. It hadn't been running at all. The radio software stopped on July 21, about six hours before I pulled the cable, and it stayed down for ten of the station's first thirteen days. Nothing told me.

Fourteen clean exits

When I sat down to plan the move on July 31, the first pass at the box was read-only, with nothing restarted. The service manager said the radio daemon had exited with status 0 fourteen times, a few seconds apart, and then stopped trying for good.

Status 0 is how a program says it finished normally. So the reasoning went like this: a dying radio chip, a failed bus or a heat problem produces crashes or timeouts, not fourteen clean exits. That pushed the suspicion toward the hardware, and the verdict toward keeping the old station as the gateway. Every step holds up except the first fact: the software had been crashing the whole time.

I wasn't ready to write off a radio on a status code

That evaluation was my AI assistant's, and it was careful work. But it rested on one number nobody had looked behind, and the radio was less than two weeks old. Rather than drop the plan, I asked for this:

try building the system on the WisMesh Station as we want it to be but turn off transmit and then monitor it

Build everything for real, so the real problems show up, but leave the station unable to put anything on the air.

Transmit was switched off by editing the radio's saved config offline, before the daemon ever started. A command after startup would have left a short window to send. Then a small read-only script spoke to the radio chip directly over its SPI bus. The chip answered with the exact value a Semtech SX1262 holds after reset, and a test value written to the register read back correctly. That is a healthy chip saying hello, and the hardware theory was done before the radio had been switched on.

What the log said once something kept it

The station had kept its system log in memory only, so the July 21 crash records were gone. With the log switched to disk, the real message was right there:

INFO  Node database full with 200 nodes. Erasing oldest entry
terminate called after throwing an instance of 'std::out_of_range'
Aborted

The station keeps a database of every node it has heard, and its config capped that at 200. On July 21 that database reached 200 known nodes. When the database is full, the software evicts the oldest entry, and that eviction code throws an error and aborts. Every restart hit the same crash.

The clean exits came from the start script that ships on the station. On the path it took here, it runs the daemon and then ends with exit 0, whatever happened. So there were two bugs stacked: a crash, and a wrapper that reported the crash as success.

The fix was small and reversible. Raising the cap from 200 to 300 brought the daemon back for the first time since July 21. Then I raised it to 2,000, because I expect this mesh to have hundreds of nodes. I raised the ceiling rather than pruning the database, because pruning throws away other nodes' keys and leaves you one growth spurt from the same crash. From then on, the service started the daemon directly instead of through the script, so a crash would show up as a crash.

The bug only an inert build could catch

Watching paid off again. Each time the daemon restarted with the old, stale node database, it gave itself a new random node ID. I saw it go through three identities in a row. Other nodes cache a node's identity and key, and keep them. With transmit on, every reboot would have pushed a fresh, permanent identity into the databases of the 1,085 key-carrying nodes my older station knew about. I couldn't have taken those back from my side.

The fix was to pin the identity with a startup flag and clear the stale database files, while keeping the file that holds the node's keypair. One trap: the help text says the --hwid flag takes a MAC address. It takes an integer, and a MAC makes it throw.

One more catch, on the simulated nodes I was setting up for the bots: setting the LoRa region quietly flipped transmit back on, unasked. No radio was attached, so nothing went out, but on a real radio that is a transmission nobody chose. From then on, every LoRa change was followed by reading transmit back on that node.

The parts that were not magic

I was the one who believed the station had been running fine. Everything that should have told me otherwise failed into a reassuring state. The service manager gave up quietly. The mesh dashboard logged 44,454 connection errors and still called itself healthy, because its health check only asked whether the dashboard was up. The job that reported the station's health had no network. And the log that held the answer was in memory.

The check I actually needed asks whether the radio is still hearing things: the newest message timestamp has to keep moving forward. A timestamp that merely exists proves nothing, because a frozen database has a perfectly good timestamp.

If you are about to try this

• Diagnose before you replace anything. A read-only register read tells a dead chip from dead software without keying the transmitter.
• Don't trust an exit code that comes through a wrapper. Read the process's own log, and have the service call the daemon directly.
• Keep logs on disk on any box you might have to debug later.
• Size the node database for the mesh you'll have, and watch how full it is.
• Build it fully but inert, then watch. Turn transmit off before the first start, and read it back after every LoRa change.

Build it yourself

What it does: a home Meshtastic base station running the Linux Meshtastic daemon on a Raspberry Pi with a LoRa board.

How the pieces fit: a service calls the daemon directly, its config sets the node database ceiling well above your mesh size, the identity is pinned, the log lives on disk, and a monitor checks that the newest heard message keeps getting newer. Transmit stays off until the rest has run cleanly.

Time: the diagnosis, the fixes and the inert build took one day.

If you have chased a "dead" radio that turned out to be software, I would like to hear about it.