P1: Mounting file system - additional checks and troubleshooting

[nanometrenat]

We have experienced a number of intermittent issues with various Mesh Extenders recently. It seems that on startup something is causing the file system to not mount as it should.

S99mountcheck needs to be fixed. Symptoms: on device 1CAFA650 in Epau (which has Nov 16 firmware version) the splash page etc. were loaded onto the root file system and should not have been. This also meant it ran out of space. This was identified on 18 Nov so Paul deleted the splash page to temporarily help the issue. On 19 Nov we saw that this device was still having problems - rebooting constantly, it also had the SSID issue intermittently (#10)

The Mount Check needs to happen BEFORE ServalD starts.

Need to ensure the mounting is reliable. Separate issue being logged for the Radio mounting.

[gardners]

LBARD now has a facility where it monitors whether the file systems are mounted, so that we can easily verify the status of this, without having to log in via ssh.

I am also contemplating how we can make a Mesh Extender function without a working USB/SD storage device, to at least a limited extent. In practice, this would likely mean that we have to run /serval and /serval-var from RAM, which would greatly restrict the size of Rhizome database that can be used. The idea would be that we would mount a RAMFS on /serval and /serval-var prior to trying to mount the USB/SD, so that the fall-back would be RAMFS, instead of the internal flash.

As side-effect would also be that such mesh extenders would change SID (and hence MeshExtender-* SSID) each reboot, as the keyring would also be stored in RAM.

Some care would be required to detect when a USB/SD can mount, but mounts read-only (or transitions to read-only due to USB bus errors or other problems. In that case, the USB/SD partitions would need to be unmounted, and the RAMFS remounted, and servald restarted, so that it starts working from the RAMFS.

[gardners]

Commit servalproject/lbard@94a0765 calls the scripts just added above. Yet to be tested on real hardware.