Ghost notifications from offline printers or private key compromised?

[caffeineflo]

Over the past 48hrs two of my printers (Creality K1s) that have mobileraker_companion installed to server notifications to my phone but were offline (literally not plugged in), started to serve me notifications of started prints and status updates.

The notifications (see screenshot) came up in the middle of the night, telling me the status of my prints, reporting at 50% and when they finished.

What's even weirder is that the files mentioned haven't existed on my systems for a long time anymore as well as the "UNKNOWN" file wasn't something my system ever had.

Generally, my printers and all of my network isn't exposed to the WWW, only a single wireguard server. Which leaves the question to me, wether my private-keys that notify the FCM service was compromised somehow or there was some old task running that notified me now?

In any case, I'd love to understand what happened here.

Thanks

[Clon1998]

Over the past 48hrs two of my printers (Creality K1s) that have mobileraker_companion installed to server notifications to my phone but were offline (literally not plugged in), started to serve me notifications of started prints and status updates.

The notifications (see screenshot) came up in the middle of the night, telling me the status of my prints, reporting at 50% and when they finished.

What's even weirder is that the files mentioned haven't existed on my systems for a long time anymore as well as the "UNKNOWN" file wasn't something my system ever had.

Generally, my printers and all of my network isn't exposed to the WWW, only a single wireguard server. Which leaves the question to me, wether my private-keys that notify the FCM service was compromised somehow or there was some old task running that notified me now?

In any case, I'd love to understand what happened here.

Thanks

Hey,
That's the first time that a user reports ghost notifications. I'd suspect that maybe some older notifications were not delivered by Apple's push servers, which finally got delivered. This can happen if your phone is offline or the energy constraints of the OS prevent the delivery. However, the lifetime of a notification should be set to 7 days, and any older unsuccessfully delivered notifications should be dropped.

In regard to your concerns about keys. The companion works by directly communicating with the backend service (via SSL), which acts as a relay between all companion instances and Firebase for notification delivery. The backend server does not store any information in regard to your push token or content. Also, the only places where your push token is permanently stored are 1. your device and 2. the companion/moonraker database running on your printer. If you are concerned, you can also entirely remove the app and reinstall it in order to receive a completely new pushToken.

As a first step, I'd suggest you clear the Notification Device Registry. This can be done by opening the Machine/Printer settings within the Mobileraker and scrolling all the way to the bottom.

Please let me know if this happens again and I will investigate this behavior more indepth.

[caffeineflo]

@Clon1998 Thanks for your reply.

Initially I was hoping that this could be closed and we’ll be good, but I haven’t turned on my printers since our last discussion, yet, I received more notifications last night while on vacation 😅

I’m out until Jan 6th, so I’ll try to look closer into this from my side after my return.
FWIW, I’m an iOS dev myself as well, so hopefully this can be painless debugging.

Thanks!

[Clon1998]

@Clon1998 Thanks for your reply.

Initially I was hoping that this could be closed and we’ll be good, but I haven’t turned on my printers since our last discussion, yet, I received more notifications last night while on vacation 😅

I’m out until Jan 6th, so I’ll try to look closer into this from my side after my return. FWIW, I’m an iOS dev myself as well, so hopefully this can be painless debugging.

Thanks!

That is weird. Enjoy your time off. Once you're back please provide me the log file of the companion. I am wondering if the companion is behaving weired.

[caffeineflo]

@Clon1998 Thanks for your reply.
Initially I was hoping that this could be closed and we’ll be good, but I haven’t turned on my printers since our last discussion, yet, I received more notifications last night while on vacation 😅
I’m out until Jan 6th, so I’ll try to look closer into this from my side after my return. FWIW, I’m an iOS dev myself as well, so hopefully this can be painless debugging.
Thanks!

That is weird. Enjoy your time off. Once you're back please provide me the log file of the companion. I am wondering if the companion is behaving weired.

FWIW, the companion is currently offline and has been for 2-3 weeks since I’m running the companion on a K1 that’s completely powered down. So this had to be a firebase stored message that was now senr out

[Clon1998]

@Clon1998 Thanks for your reply.
Initially I was hoping that this could be closed and we’ll be good, but I haven’t turned on my printers since our last discussion, yet, I received more notifications last night while on vacation 😅
I’m out until Jan 6th, so I’ll try to look closer into this from my side after my return. FWIW, I’m an iOS dev myself as well, so hopefully this can be painless debugging.
Thanks!

That is weird. Enjoy your time off. Once you're back please provide me the log file of the companion. I am wondering if the companion is behaving weired.

FWIW, the companion is currently offline and has been for 2-3 weeks since I’m running the companion on a K1 that’s completely powered down. So this had to be a firebase stored message that was now senr out

Ohh okay. Once I have the log, I might be able to have a look at the backend logs to see exactly what the reason is. I don't think that the notifications are held back for so long tho. Also, there should never be a collision of Firebase cloud messaging IDs, as far as I know.

[caffeineflo]

Hey @Clon1998!

I'm back and wanted to give an update. Since we last spoke over a week ago, I started getting more and more notifications that were consistent with someone printing. Initially I thought these might still be some of my old queued up notifications, until it became clear that I must have been receiving someone elses notifications. They started printing XBox controller stuff and I never owned an Xbox (Playstation kid ...).
Attached is a screenshot of a bunch of notifications I've gotten:

One of the notifications made me think though: It became clear the persons notifications I was receiving were also coming from another Creality K1 printer and the chances he used the same method to install mobileraker_companion as I did (via https://github.com/Guilouz/Creality-K1-and-K1-Max) are actually quite high.
I knew from their repo and past problems I had with the pyenv they are shipping (which is just a bundled up precompiled pyenv that is ~1 year old) that there might be some issues with that and incompatibilities since the dependencies don't fully match both moonraker and mobileraker_companion anymore (which I have already started to fix separately of this issue).

So this got me thinking if this might be a problem where there are either key collisions happening due to a large number of K1 users using the same scripts to install mobileraker_companion or something else related to that setup that might be independent of mobileraker_companion?
If you have an idea, I'm all ears! In any case, I started looking through the source code of this repo and I'm linking the relevant portion of the install steps that are used on pretty much all K1s to install mobileraker_companion here: https://github.com/Guilouz/Creality-K1-and-K1-Max/blob/main/Scripts/installer.sh#L1847

Sorry if I end(ed) up causing you undue stress with this, I'll also add my logs in a second after I booted up my printers

[caffeineflo]

Here's the log from the last time the printer successfully printed something
mobileraker.log

[Clon1998]

@caffeineflo, no worries, and thanks for the investigation.
The logs look good, but as you mentioned, it seems like someone else is using your FCM token.
In my infinite wisdom, I removed any reference to the token from the log files since users tended to just upload log files without removing them, which could potentially be abused. Unfortunately, this means that I am unable to check the backend server for any relevant information. However, I doubt that I would discover anything new even if I could.

I currently do not store any of the FCM Tokens in Firebase or my backend (That acts as a relay between all Companions and Firebase Cloud Messaging/APNs for push notifications) since I expected that Firebase would prevent any token collisions in the first place. Given that I just have roughly 15k users, a token collision was very unlikely (Congrats I guess?).

Regarding the installation of the K1 script: since the token is generated by your phone or tablet and stored in moonraker's database, any problems with the token are not related to the installation script.

For now, I would suggest you remove the app entirely and install it once again to ensure firebase generates a new FCM Token for your device. After adding the printer again, I suggest you clear the FCM token repository which can be done at the very bottom of the printer settings page.