Skip to content

Commit 4a78c01

Browse files
yozhao101yozhao101
authored
[config] Enable/disable container monitoring when starting/stopping the services. (#1471)
What I did When we ran the command sudo config load, sudo config reload or sudo config load_minigraph, the containers swss, snmp, lldp, teamd, syncd, snmp, bgp, radv, pmon, dhcp_relay, telemetry and restapi would be stopped and then restarted. The script container_checker ran by Monit will generate false alerting messages into syslog to indicate some containers were not running during such stopping and restarting process. So this PR aims to prevent Monit from generating false alarm messages. How I did it Before stopping services, we disable Monit to monitor the running status of containers. After restarting services, we enable Monit to monitor the running status of containers again. How to verify it I deliberately reduce the monitoring interval of Monit from 60 seconds to 10 seconds to ensure the alerting messages from the script container_checker was generated during sudo config reload, sudo config load and sudo config load_minigraph. After this change was added into _stop_services(...) and _restart_services(...) , I checked that the alerting messages from container_checker did not appear in the syslog. I verified this change on the device str-a7050-acs-3. Previous command output (if the output of a command-line utility has changed) admin@vlab-01:~$ sudo config reload -y Executing stop of service telemetry... Warning: Stopping telemetry.service, but it can still be activated by: telemetry.timer Executing stop of service swss... Executing stop of service lldp... Executing stop of service pmon... Executing stop of service bgp... Running command: /usr/local/bin/sonic-cfggen -j /etc/sonic/init_cfg.json -j /etc/sonic/config_db.json --write-to-db Running command: /usr/local/bin/db_migrator.py -o migrate Executing reset-failed of service bgp... Executing reset-failed of service dhcp_relay... Executing reset-failed of service hostname-config... Executing reset-failed of service interfaces-config... Executing reset-failed of service lldp... Executing reset-failed of service ntp-config... Executing reset-failed of service pmon... Executing reset-failed of service radv... Executing reset-failed of service rsyslog-config... Executing reset-failed of service snmp... Executing reset-failed of service swss... Executing reset-failed of service syncd... Executing reset-failed of service teamd... Executing reset-failed of service telemetry... Executing restart of service hostname-config... Executing restart of service interfaces-config... Executing restart of service ntp-config... Executing restart of service rsyslog-config... Executing restart of service swss... Executing restart of service bgp... Executing restart of service pmon... Executing restart of service lldp... Executing restart of service telemetry... Reloading Monit configuration ... Reinitializing monit daemon New command output (if the output of a command-line utility has changed) admin@vlab-01:~$ sudo config reload -y Disabling container monitoring ... Executing stop of service telemetry... Warning: Stopping telemetry.service, but it can still be activated by: telemetry.timer Executing stop of service swss... Executing stop of service lldp... Executing stop of service pmon... Executing stop of service bgp... Running command: /usr/local/bin/sonic-cfggen -j /etc/sonic/init_cfg.json -j /etc/sonic/config_db.json --write-to-db Running command: /usr/local/bin/db_migrator.py -o migrate Executing reset-failed of service bgp... Executing reset-failed of service dhcp_relay... Executing reset-failed of service hostname-config... Executing reset-failed of service interfaces-config... Executing reset-failed of service lldp... Executing reset-failed of service ntp-config... Executing reset-failed of service pmon... Executing reset-failed of service radv... Executing reset-failed of service rsyslog-config... Executing reset-failed of service snmp... Executing reset-failed of service swss... Executing reset-failed of service syncd... Executing reset-failed of service teamd... Executing reset-failed of service telemetry... Executing restart of service hostname-config... Executing restart of service interfaces-config... Executing restart of service ntp-config... Executing restart of service rsyslog-config... Executing restart of service swss... Executing restart of service bgp... Executing restart of service pmon... Executing restart of service lldp... Executing restart of service telemetry... Enabling container monitoring ... Reloading Monit configuration ... Reinitializing monit daemon
1 parent dd3c2c3 commit 4a78c01

File tree

2 files changed

+9
-1
lines changed

2 files changed

+9
-1
lines changed

‎config/main.py

+6
Original file line numberDiff line numberDiff line change
@@ -669,6 +669,9 @@ def _get_disabled_services_list(config_db):
669669

670670

671671
def _stop_services():
672+
click.echo("Disabling container monitoring ...")
673+
clicommon.run_command("sudo monit unmonitor container_checker")
674+
672675
click.echo("Stopping SONiC target ...")
673676
clicommon.run_command("sudo systemctl stop sonic.target")
674677

@@ -692,6 +695,9 @@ def _restart_services():
692695
click.echo("Reloading Monit configuration ...")
693696
clicommon.run_command("sudo monit reload")
694697

698+
click.echo("Enabling container monitoring ...")
699+
clicommon.run_command("sudo monit monitor container_checker")
700+
695701

696702
def interface_is_in_vlan(vlan_member_table, interface_name):
697703
""" Check if an interface is in a vlan """

‎tests/config_test.py

+3-1
Original file line numberDiff line numberDiff line change
@@ -12,12 +12,14 @@
1212
from utilities_common.db import Db
1313

1414
load_minigraph_command_output="""\
15+
Disabling container monitoring ...
1516
Stopping SONiC target ...
1617
Running command: /usr/local/bin/sonic-cfggen -H -m --write-to-db
1718
Running command: pfcwd start_default
1819
Running command: config qos reload --no-dynamic-buffer
1920
Restarting SONiC target ...
2021
Reloading Monit configuration ...
22+
Enabling container monitoring ...
2123
Please note setting loaded from minigraph will be lost after system reboot. To preserve setting, run `config save`.
2224
"""
2325

@@ -49,7 +51,7 @@ def test_load_minigraph(self, get_cmd_module, setup_single_broadcom_asic):
4951
traceback.print_tb(result.exc_info[2])
5052
assert result.exit_code == 0
5153
assert "\n".join([l.rstrip() for l in result.output.split('\n')]) == load_minigraph_command_output
52-
assert mock_run_command.call_count == 7
54+
assert mock_run_command.call_count == 9
5355

5456
@classmethod
5557
def teardown_class(cls):

0 commit comments

Comments
 (0)