Test failes when using negativ values in delta operator

[chuegel]

Description of Issue/Question

Delta values are not calculated correctly when using negative values e.g. -10% or postive values e.g +10%
It looks like delta tests are behaving just like when using only the percentage 10%

Setup

(Please provide relevant configs, xml snapshots and related files (Be sure to remove sensitive info).)

Pre snapshot:

<route-summary-information>
<as-number>44745</as-number>
<router-id>10.255.255.250</router-id>
<route-table>
<table-name>inet.0</table-name>
<destination-count>141</destination-count>
<total-route-count>141</total-route-count>
<active-route-count>100</active-route-count>
<holddown-route-count>1</holddown-route-count>
<hidden-route-count>1</hidden-route-count>
<protocols>
<protocol-name>Direct</protocol-name>
<protocol-route-count>9</protocol-route-count>
<active-route-count>8</active-route-count>
</protocols>
<protocols>
<protocol-name>Local</protocol-name>
<protocol-route-count>7</protocol-route-count>
<active-route-count>7</active-route-count>
</protocols>
<protocols>
<protocol-name>OSPF</protocol-name>
<protocol-route-count>124</protocol-route-count>
<active-route-count>123</active-route-count>
</protocols>
<protocols>
<protocol-name>Static</protocol-name>
<protocol-route-count>1</protocol-route-count>
<active-route-count>1</active-route-count>
</protocols>
</route-table>
</route-summary-information>

Post snapshot:

<route-summary-information>
<as-number>44745</as-number>
<router-id>10.255.255.250</router-id>
<route-table>
<table-name>inet.0</table-name>
<destination-count>140</destination-count>
<total-route-count>140</total-route-count>
<active-route-count>111</active-route-count>
<holddown-route-count>0</holddown-route-count>
<hidden-route-count>1</hidden-route-count>
<protocols>
<protocol-name>Direct</protocol-name>
<protocol-route-count>9</protocol-route-count>
<active-route-count>8</active-route-count>
</protocols>
<protocols>
<protocol-name>Local</protocol-name>
<protocol-route-count>7</protocol-route-count>
<active-route-count>7</active-route-count>
</protocols>
<protocols>
<protocol-name>OSPF</protocol-name>
<protocol-route-count>123</protocol-route-count>
<active-route-count>123</active-route-count>
</protocols>
<protocols>
<protocol-name>Static</protocol-name>
<protocol-route-count>1</protocol-route-count>
<active-route-count>1</active-route-count>
</protocols>
</route-table>
</route-summary-information>

Testfile:

tests_include:
  - check_route_summary

check_route_summary:
- command: show route summary
- iterate:
    xpath: //route-summary-information/route-table
    id: ./table-name
    ignore-null: true
    tests:
      - delta: active-route-count, +10%
        info: "Active routes on: <{{post['table-name']}}> has not changed by more than 10%. Pre: <{{pre['active-route-count']}}>, Post <{{post['active-route-count']}}> "
        err: "WARNING!! <{{post['table-name']}}> has changed by more than 10% delta. Pre <{{pre['active-route-count']}}>, Post <{{post['active-route-count']}}> "
      - delta: active-route-count, -10%
        info: "Active routes on: <{{post['table-name']}}> has not changed by more than 10%. Pre: <{{pre['active-route-count']}}>, Post <{{post['active-route-count']}}> "
        err: "WARNING!! <{{post['table-name']}}> has changed by more than 10% delta. Pre <{{pre['active-route-count']}}>, Post <{{post['active-route-count']}}> "

Config file:

hosts:
  - device: 10.1.1.1
    username : ansible
tests:
   - test_routes-2.yml 

Test result:

jsnapy --port 22 --check pre post -f jsnapy-host.cfg -v

jsnapy.cfg file location used : /etc/jsnapy
Configuration file location used : /etc/jsnapy
*************************** Device: 10.1.1.1 ***************************
Tests Included: check_route_summary 
************************ Command: show route summary ************************
-----------------------Performing delta Test Operation-----------------------
WARNING!! <inet.0> has changed by more than 10% delta. Pre <100.0>, Post <111.0> 
FAIL | All "active-route-count" is not with in delta difference of +10% [ 0 value matched / 1 value failed ]
-----------------------Performing delta Test Operation-----------------------
WARNING!! <inet.0> has changed by more than 10% delta. Pre <100.0>, Post <111.0> 
FAIL | All "active-route-count" is not with in delta difference of -10% [ 0 value matched / 1 value failed ]
------------------------------- Final Result!! -------------------------------
check_route_summary : Failed
Total No of tests passed: 0
Total No of tests failed: 2 
Overall Tests failed!!! 

As you see, both tests are failing instead of just one which has the delta set to +10%

If I change the <active-route-count> in the post snapshot to <active-route-count>89</active-route-count> e.g -10% of the pre value, both tests fail again:

jsnapy --port 22 --check pre  post -f jsnapy-host.cfg -v
jsnapy.cfg file location used : /etc/jsnapy
Configuration file location used : /etc/jsnapy
*************************** Device: 10.1.1.1 ***************************
Tests Included: check_route_summary 
************************ Command: show route summary ************************
-----------------------Performing delta Test Operation-----------------------
WARNING!! <inet.0> has changed by more than 10% delta. Pre <100.0>, Post <89.0> 
FAIL | All "active-route-count" is not with in delta difference of +10% [ 0 value matched / 1 value failed ]
-----------------------Performing delta Test Operation-----------------------
WARNING!! <inet.0> has changed by more than 10% delta. Pre <100.0>, Post <89.0> 
FAIL | All "active-route-count" is not with in delta difference of -10% [ 0 value matched / 1 value failed ]
------------------------------- Final Result!! -------------------------------
check_route_summary : Failed
Total No of tests passed: 0
Total No of tests failed: 2 
Overall Tests failed!!! 

When using following testfile with only 10%:

tests_include:
  - check_route_summary

check_route_summary:
- command: show route summary
- iterate:
    xpath: //route-summary-information/route-table
    id: ./table-name
    ignore-null: true
    tests:
      - delta: active-route-count, 10%
        info: "Active routes on: <{{post['table-name']}}> has not changed by more than 10%. Pre: <{{pre['active-route-count']}}>, Post <{{post['active-route-count']}}> "
        err: "WARNING!! <{{post['table-name']}}> has changed by more than 10% delta. Pre <{{pre['active-route-count']}}>, 

the test fails for every value lower/higher then 10% from the pre snapshot value (here 100)

Steps to Reproduce Issue

(Include debug logs if possible and relevant. Error trace would be helpful too)

see above

Versions Report

(Provided by running jsnapy --version. Please also mention python version.)

JSNAPy version: 1.3.6

[chidanandpujar]

Hi ,

Could you please check and let me know , if following functionality works according to your requirements .

documentation -
https://www.juniper.net/documentation/us/en/software/junos-snapshot/snapshot-administrator/topics/ref/script/automation-junos-snapshot-operator-delta.html

the delta function is as per below .
Absolute percentage -
if pre-snapshot is 100 and post-snapshot is 110 and delta is 10%
delta check will Pass.

if pre-snapshot is 100 and post-snapshot value 90 and delta is 10%
delta check will Pass .

if post-snapthot is 111 or 89 , delta check will Fail.

Positive percentage -
if pre-snapshot is 100 and post-snapshot value is within the range of 100 and 110 and delta is +10%
delta check will Pass

Negative percentage -

if pre-snapstho is 100 and post-snapshot value is within the range of 100 to 90 and delta is -10%
delta check will Pass

Thanks & Regards
Chidanand

[jaakub]

Hi,

I'm seeing similar issues when using negative percentages in JSNAPy version: 1.3.6.

physical_interfaces_ae:
  - command: show interfaces ae*
  - iterate:
      xpath: //physical-interface
      id: ./name
      tests:
        - delta: ./traffic-statistics/input-bps, -25%
          err: "{{ id_0 }} current input bps of {{ post['./traffic-statistics/input-bps'] }} bps has decreased by more than 25%. Previous value is {{ pre['./traffic-statistics/input-bps'] }} bps."
        - delta: ./traffic-statistics/output-bps, -25%
          err: "{{ id_0 }} current output bps of {{ post['./traffic-statistics/output-bps'] }} bps has decreased by more than 25%. Previous value is {{ pre['./traffic-statistics/output-bps'] }} bps."

As you can see, I only expect to see an error message when the post value decreases by 25% from pre-value, however, this does not seem to be the case, and error is still raised even if the post value has increased by more than 25%. As I understand, this should not be raised as an error.

['ae12'] current output bps of 2368033344.0 bps has decreased by more than 25%. The previous value is 2122350472.0 bps.
['ae22'] current output bps of 105692104.0 bps has decreased by more than 25%. The previous value is 97309928.0 bps.
['ae6'] current output bps of 2976.0 bps has decreased by more than 25%. The previous value is 2912.0 bps.

I have added comments to the #392 .

[chidanandpujar]

Hi,

I have updated the fix and verified , please find the results .

Case 1: When pre-snapshot value is 100 and post-snanpshot is +10% i.e 111

:~/jsnapy_issue_392/jsnapy/samples# cat /root/jsnapy_issue_392/venv/etc/jsnapy/snapshots/10.220.12.56_22_pre_show_route_summary.xml
<route-summary-information>
	<as-number>44745</as-number>
	<router-id>10.255.255.250</router-id>
	<route-table>
		<table-name>inet.0</table-name>
		<destination-count>141</destination-count>
		<total-route-count>141</total-route-count>
		<active-route-count>100</active-route-count>
		<holddown-route-count>1</holddown-route-count>
		<hidden-route-count>1</hidden-route-count>
		<protocols>
			<protocol-name>Direct</protocol-name>
			<protocol-route-count>9</protocol-route-count>
			<active-route-count>8</active-route-count>
		</protocols>
		<protocols>
			<protocol-name>Local</protocol-name>
			<protocol-route-count>7</protocol-route-count>
			<active-route-count>7</active-route-count>
		</protocols>
		<protocols>
			<protocol-name>OSPF</protocol-name>
			<protocol-route-count>124</protocol-route-count>
			<active-route-count>123</active-route-count>
		</protocols>
		<protocols>
			<protocol-name>Static</protocol-name>
			<protocol-route-count>1</protocol-route-count>
			<active-route-count>1</active-route-count>
		</protocols>
	</route-table>
</route-summary-information>

~/jsnapy_issue_392/jsnapy/samples# cat /root/jsnapy_issue_392/venv/etc/jsnapy/snapshots/10.220.12.56_22_post_show_route_summary.xml
<route-summary-information>
	<as-number>44745</as-number>
	<router-id>10.255.255.250</router-id>
	<route-table>
		<table-name>inet.0</table-name>
		<destination-count>140</destination-count>
		<total-route-count>140</total-route-count>
		<active-route-count>111</active-route-count>
		<holddown-route-count>0</holddown-route-count>
		<hidden-route-count>1</hidden-route-count>
		<protocols>
			<protocol-name>Direct</protocol-name>
			<protocol-route-count>9</protocol-route-count>
			<active-route-count>8</active-route-count>
		</protocols>
		<protocols>
			<protocol-name>Local</protocol-name>
			<protocol-route-count>7</protocol-route-count>
			<active-route-count>7</active-route-count>
		</protocols>
		<protocols>
			<protocol-name>OSPF</protocol-name>
			<protocol-route-count>123</protocol-route-count>
			<active-route-count>123</active-route-count>
		</protocols>
		<protocols>
			<protocol-name>Static</protocol-name>
			<protocol-route-count>1</protocol-route-count>
			<active-route-count>1</active-route-count>
		</protocols>
	</route-table>
</route-summary-information>
                                                                                                                                                                                  

:~/jsnapy_issue_392/jsnapy/samples# cat test.yaml 
tests_include:
  - check_route_summary

check_route_summary:
- command: show route summary
- iterate:
    xpath: //route-summary-information/route-table
    id: ./table-name
    ignore-null: true
    tests:
      - delta: active-route-count, +10%
        info: "Active routes on: <{{post['table-name']}}> has not changed by more than 10%. Pre: <{{pre['active-route-count']}}>, Post <{{post['active-route-count']}}> "
        err: "WARNING!! <{{post['table-name']}}> has changed by more than 10% delta. Pre <{{pre['active-route-count']}}>, Post <{{post['active-route-count']}}> "
      - delta: active-route-count, -10%
        info: "Active routes on: <{{post['table-name']}}> has not changed by more than 10%. Pre: <{{pre['active-route-count']}}>, Post <{{post['active-route-count']}}> "
        err: "WARNING!! <{{post['table-name']}}> has changed by more than 10% delta. Pre <{{pre['active-route-count']}}>, Post <{{post['active-route-count']}}> "


~/jsnapy_issue_392/jsnapy/samples# jsnapy --check pre post -f config.yaml
**************************** Device: 10.220.12.56 ****************************
Tests Included: check_route_summary 
************************ Command: show route summary ************************
WARNING!! <inet.0> has changed by more than 10% delta. Pre <100.0>, Post <111.0> 
FAIL | All "active-route-count" is not with in delta difference of +10% [ 0 value matched / 1 value failed ]
PASS | All "active-route-count" is with in delta difference of -10% [ 1 value matched ]
------------------------------- Final Result!! -------------------------------
check_route_summary : Failed
Total No of tests passed: 1
Total No of tests failed: 1 
Overall Tests failed!!! 




Case 2:  pre-snapshot value is 100 and post-snapshot vlaue is -10% i.e 89 

~/jsnapy_issue_392/jsnapy/samples# cat ~/jsnapy_issue_392/venv/etc/jsnapy/snapshots/10.220.12.56_22_post_show_route_summary.xml
<route-summary-information>
	<as-number>44745</as-number>
	<router-id>10.255.255.250</router-id>
	<route-table>
		<table-name>inet.0</table-name>
		<destination-count>140</destination-count>
		<total-route-count>140</total-route-count>
		<active-route-count>89</active-route-count>
		<holddown-route-count>0</holddown-route-count>
		<hidden-route-count>1</hidden-route-count>
		<protocols>
			<protocol-name>Direct</protocol-name>
			<protocol-route-count>9</protocol-route-count>
			<active-route-count>8</active-route-count>
		</protocols>
		<protocols>
			<protocol-name>Local</protocol-name>
			<protocol-route-count>7</protocol-route-count>
			<active-route-count>7</active-route-count>
		</protocols>
		<protocols>
			<protocol-name>OSPF</protocol-name>
			<protocol-route-count>123</protocol-route-count>
			<active-route-count>123</active-route-count>
		</protocols>
		<protocols>
			<protocol-name>Static</protocol-name>
			<protocol-route-count>1</protocol-route-count>
			<active-route-count>1</active-route-count>
		</protocols>
	</route-table>
</route-summary-information>



~/jsnapy_issue_392/jsnapy/samples# jsnapy --check pre post -f config.yaml
**************************** Device: 10.220.12.56 ****************************
Tests Included: check_route_summary 
************************ Command: show route summary ************************
PASS | All "active-route-count" is with in delta difference of +10% [ 1 value matched ]
WARNING!! <inet.0> has changed by more than 10% delta. Pre <100.0>, Post <89.0> 
FAIL | All "active-route-count" is not with in delta difference of -10% [ 0 value matched / 1 value failed ]
------------------------------- Final Result!! -------------------------------
check_route_summary : Failed
Total No of tests passed: 1
Total No of tests failed: 1 
Overall Tests failed!!! 



Case 3: pre-snapshot is 100 and post-snapshot is either + or - 10% ,  ie  110 or 90 .

~/jsnapy_issue_392/jsnapy/samples# cat test.yaml 
tests_include:
  - check_route_summary

check_route_summary:
- command: show route summary
- iterate:
    xpath: //route-summary-information/route-table
    id: ./table-name
    ignore-null: true
    tests:
      - delta: active-route-count, 10%
        info: "Active routes on: <{{post['table-name']}}> has not changed by more than 10%. Pre: <{{pre['active-route-count']}}>, Post <{{post['active-route-count']}}> "
        err: "WARNING!! <{{post['table-name']}}> has changed by more than 10% delta. Pre <{{pre['active-route-count']}}>, Post <{{post['active-route-count']}}> "



~/jsnapy_issue_392/jsnapy/samples# vi ~/jsnapy_issue_392/venv/etc/jsnapy/snapshots/10.220.12.56_22_post_show_route_summary.xml
<route-summary-information>
        <as-number>44745</as-number>
        <router-id>10.255.255.250</router-id>
        <route-table>
                <table-name>inet.0</table-name>
                <destination-count>140</destination-count>
                <total-route-count>140</total-route-count>
                <active-route-count>90</active-route-count>
                <holddown-route-count>0</holddown-route-count>
                <hidden-route-count>1</hidden-route-count>
                <protocols>
                        <protocol-name>Direct</protocol-name>
                        <protocol-route-count>9</protocol-route-count>
                        <active-route-count>8</active-route-count>
                </protocols>
                <protocols>
                        <protocol-name>Local</protocol-name>
                        <protocol-route-count>7</protocol-route-count>
                        <active-route-count>7</active-route-count>
                </protocols>
                <protocols>
                        <protocol-name>OSPF</protocol-name>
                        <protocol-route-count>123</protocol-route-count>
                        <active-route-count>123</active-route-count>
                </protocols>
                <protocols>
                        <protocol-name>Static</protocol-name>
                        <protocol-route-count>1</protocol-route-count>
                        <active-route-count>1</active-route-count>
                </protocols>
        </route-table>
</route-summary-information>

                                                                                      
~/jsnapy_issue_392/jsnapy/samples# jsnapy --check pre post -f config.yaml
**************************** Device: 10.220.12.56 ****************************
Tests Included: check_route_summary 
************************ Command: show route summary ************************
PASS | All "active-route-count" is with in delta difference of 10% [ 1 value matched ]
------------------------------- Final Result!! -------------------------------
check_route_summary : Passed
Total No of tests passed: 1
Total No of tests failed: 0 
Overall Tests passed!!! 


~/jsnapy_issue_392/jsnapy/samples# vi ~/jsnapy_issue_392/venv/etc/jsnapy/snapshots/10.220.12.56_22_post_show_route_summary.xml
<route-summary-information>
        <as-number>44745</as-number>
        <router-id>10.255.255.250</router-id>
        <route-table>
                <table-name>inet.0</table-name>
                <destination-count>140</destination-count>
                <total-route-count>140</total-route-count>
                <active-route-count>110</active-route-count>
                <holddown-route-count>0</holddown-route-count>
                <hidden-route-count>1</hidden-route-count>
                <protocols>
                        <protocol-name>Direct</protocol-name>
                        <protocol-route-count>9</protocol-route-count>
                        <active-route-count>8</active-route-count>
                </protocols>
                <protocols>
                        <protocol-name>Local</protocol-name>
                        <protocol-route-count>7</protocol-route-count>
                        <active-route-count>7</active-route-count>
                </protocols>
                <protocols>
                        <protocol-name>OSPF</protocol-name>
                        <protocol-route-count>123</protocol-route-count>
                        <active-route-count>123</active-route-count>
                </protocols>
                <protocols>
                        <protocol-name>Static</protocol-name>
                        <protocol-route-count>1</protocol-route-count>
                        <active-route-count>1</active-route-count>
                </protocols>
        </route-table>
</route-summary-information>
                                                                                                          

                                                                                    
~/jsnapy_issue_392/jsnapy/samples# jsnapy --check pre post -f config.yaml
**************************** Device: 10.220.12.56 ****************************
Tests Included: check_route_summary 
************************ Command: show route summary ************************
PASS | All "active-route-count" is with in delta difference of 10% [ 1 value matched ]
------------------------------- Final Result!! -------------------------------
check_route_summary : Passed
Total No of tests passed: 1
Total No of tests failed: 0 
Overall Tests passed!!! 




Case 4: when both pre-snapshot  is 100 and post-snapshot value are same i.e 100


 ~/jsnapy_issue_392/venv/etc/jsnapy/snapshots/10.220.12.56_22_post_show_route_summary.xml
<route-summary-information>
        <as-number>44745</as-number>
        <router-id>10.255.255.250</router-id>
        <route-table>
                <table-name>inet.0</table-name>
                <destination-count>140</destination-count>
                <total-route-count>140</total-route-count>
                <active-route-count>100</active-route-count>
                <holddown-route-count>0</holddown-route-count>
                <hidden-route-count>1</hidden-route-count>
                <protocols>
                        <protocol-name>Direct</protocol-name>
                        <protocol-route-count>9</protocol-route-count>
                        <active-route-count>8</active-route-count>
                </protocols>
                <protocols>
                        <protocol-name>Local</protocol-name>
                        <protocol-route-count>7</protocol-route-count>
                        <active-route-count>7</active-route-count>
                </protocols>
                <protocols>
                        <protocol-name>OSPF</protocol-name>
                        <protocol-route-count>123</protocol-route-count>
                        <active-route-count>123</active-route-count>
                </protocols>
                <protocols>
                        <protocol-name>Static</protocol-name>
                        <protocol-route-count>1</protocol-route-count>
                        <active-route-count>1</active-route-count>
                </protocols>
        </route-table>
</route-summary-information>
                                                                                                                                                                                                                          


                                                                                    
~/jsnapy_issue_392/jsnapy/samples# jsnapy --check pre post -f config.yaml
**************************** Device: 10.220.12.56 ****************************
Tests Included: check_route_summary 
************************ Command: show route summary ************************
PASS | All "active-route-count" is with in delta difference of 10% [ 1 value matched ]
------------------------------- Final Result!! -------------------------------
check_route_summary : Passed
Total No of tests passed: 1
Total No of tests failed: 0 
Overall Tests passed!!! 

Thanks & Regards
Chidanand

[chidanandpujar]

Fixed issue as port of fix #396

[dineshbaburam91]

Fixed #396