[log] Error in syslog seen on boot Failed to add device '/dev/sda3' to watch: Operation not permitted

[alexrallen]

Description

Error seen in log sometimes on boot

 Apr 16 22:44:54.796302 r-spider-05 ERR systemd-udevd[304]: sda3: Failed to add device '/dev/sda3' to watch: Operation not permitted

Steps to reproduce the issue:

1. Reboot the switch

Describe the results you received:

Error in log

Describe the results you expected:

No error log

Output of show version:

Seen on 202111

Additional information you deem important (e.g. issue happens only occasionally):

Reproduces very infrequently.

Seems to be related to the following file on the switches
/usr/lib/udev/rules.d/60-persistent-storage-dm.rules

The error seems to be coming from the following flow in systemd-udevd
https://github.com/systemd/systemd/blob/v247/src/udev/udev-watch.c#L98

Not sure why the inotify_watch syscall is failing with Operation not permitted there is no documentation supporting why this could be the case.
https://man7.org/linux/man-pages/man2/inotify_add_watch.2.html

Attached below is a techsupport of a switch which reproduced this error log
sonic_dump_r-bulldog-04_20220601_030440.tar.gz
log.txt

[yxieca]

@saiarcot895 are you able to take a look at this?

[Junchao-Mellanox]

Could it be the same issue of systemd/systemd#24668 ?

[Junchao-Mellanox]

A potential fix: #15253