Minor SRX Pipeline issue - Addition of junos@MIB breaks GROK

[novaksam]

• Version: 8.8.2
• Operating System: Centos 8 Stream

For the example log: <14>1 2023-08-08T14:28:00.778-05:00 Route1- RT_FLOW - RT_FLOW_SESSION_DENY [[email protected] source-address="192.168.1.1" source-port="39017" destination-address="8.8.4.4" destination-port="53" connection-tag="0" service-name="junos-dns-udp" protocol-id="17" icmp-type="0" policy-name="dns_deny_outbound" source-zone-name="trust" destination-zone-name="untrust" application="UNKNOWN" nested-application="UNKNOWN" username="N/A" roles="N/A" packet-incoming-interface="reth0.0" encrypted="No" reason="Denied by policy" session-id="85905209174" application-category="N/A" application-sub-category="N/A" application-risk="-1" application-characteristics="N/A" src-vrf-grp="N/A" dst-vrf-grp="N/A"]

The initial grok processor on the juniper-srx-pipeline pipeline strips out the source address. Addition of an option EMAILADDRESS pattern resolves the issue.

"grok": {
        "field": "message",
        "patterns": [
              "^<%{POSINT:syslog_pri}>(\\d{1,3}\\s)?(?:%{TIMESTAMP_ISO8601:_temp_.raw_date})\\s%{SYSLOGHOST:syslog_hostname}\\s%{PROG:syslog_program}\\s(?:%{POSINT:syslog_pid}|-)?\\s%{WORD:log_type}\\s\\[\\s?%{GREEDYDATA:log.original}\\]$"
        ]
      }

Snippet of event.original:

"original": "[email protected] source-address=\"192.168.1.1\" source-port=\"39017\"

Modifying \\s?%{GREEDYDATA:log.original} to \\s?(%{EMAILADDRESS} )?%{GREEDYDATA:log.original} allows the source.address to be correctly collected and enriched

"grok": {
        "field": "message",
        "patterns": [
          "^<%{POSINT:syslog_pri}>(\\d{1,3}\\s)?(?:%{TIMESTAMP_ISO8601:_temp_.raw_date})\\s%{SYSLOGHOST:syslog_hostname}\\s%{PROG:syslog_program}\\s(?:%{POSINT:syslog_pid}|-)?\\s%{WORD:log_type}\\s\\[\\s?(%{EMAILADDRESS} )?%{GREEDYDATA:log.original}\\]$"
        ]
      }

Snippet of event.original:

"original": "source-address=\"192.168.1.1\" source-port=\"39017\"

[elasticmachine]

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

[efd6]

I am unable to reproduce this on tip or v8.9.2. This is the event that is produced using the event pasted in the issue.

{
    "@timestamp": "2023-08-08T17:28:00.778-02:00",
    "client.ip": "192.168.1.1",
    "client.port": 39017,
    "destination.ip": "8.8.4.4",
    "destination.port": 53,
    "event.action": "flow_deny",
    "event.category": [
        "network"
    ],
    "event.dataset": "juniper.srx",
    "event.kind": "event",
    "event.module": "juniper",
    "event.original": "source-address=\"192.168.1.1\" source-port=\"39017\" destination-address=\"8.8.4.4\" destination-port=\"53\" connection-tag=\"0\" service-name=\"junos-dns-udp\" protocol-id=\"17\" icmp-type=\"0\" policy-name=\"dns_deny_outbound\" source-zone-name=\"trust\" destination-zone-name=\"untrust\" application=\"UNKNOWN\" nested-application=\"UNKNOWN\" username=\"N/A\" roles=\"N/A\" packet-incoming-interface=\"reth0.0\" encrypted=\"No\" reason=\"Denied by policy\" session-id=\"85905209174\" application-category=\"N/A\" application-sub-category=\"N/A\" application-risk=\"-1\" application-characteristics=\"N/A\" src-vrf-grp=\"N/A\" dst-vrf-grp=\"N/A\"",
    "event.outcome": "success",
    "event.risk_score": -1.0,
    "event.severity": 14,
    "event.timezone": "-02:00",
    "event.type": [
        "connection",
        "denied"
    ],
    "fileset.name": "srx",
    "input.type": "log",
    "juniper.srx.connection_tag": "0",
    "juniper.srx.encrypted": "No",
    "juniper.srx.icmp_type": "0",
    "juniper.srx.process": "RT_FLOW",
    "juniper.srx.reason": "Denied by policy",
    "juniper.srx.service_name": "junos-dns-udp",
    "juniper.srx.session_id": "85905209174",
    "juniper.srx.tag": "RT_FLOW_SESSION_DENY",
    "log.level": "informational",
    "log.offset": 20538,
    "network.iana_number": "17",
    "observer.egress.zone": "untrust",
    "observer.ingress.interface.name": "reth0.0",
    "observer.ingress.zone": "trust",
    "observer.name": "Route1-",
    "observer.product": "SRX",
    "observer.type": "firewall",
    "observer.vendor": "Juniper",
    "related.ip": [
        "192.168.1.1",
        "8.8.4.4"
    ],
    "rule.name": "dns_deny_outbound",
    "server.ip": "8.8.4.4",
    "server.port": 53,
    "service.type": "juniper",
    "source.ip": "192.168.1.1",
    "source.port": 39017,
    "tags": [
        "forwarded",
        "juniper.srx"
    ]
}

If I make the following change,

diff --git a/x-pack/filebeat/module/juniper/srx/test/flow.log b/x-pack/filebeat/module/juniper/srx/test/flow.log
index 16ce15c66f..857efb6a70 100644
--- a/x-pack/filebeat/module/juniper/srx/test/flow.log
+++ b/x-pack/filebeat/module/juniper/srx/test/flow.log
@@ -26,1 +26,1 @@
 <14>1 2020-07-13T16:43:05.041Z SRX100HM RT_FLOW - RT_FLOW_SESSION_CLOSE [[email protected] reason="idle Timeout" source-address="10.1.1.100" source-port="64720" destination-address="91.228.167.172" destination-port="8883" connection-tag="0" service-name="None" nat-source-address="172.19.34.100" nat-source-port="24519" nat-destination-address="91.228.167.172" nat-destination-port="8883" nat-connection-tag="0" src-nat-rule-type="source rule" src-nat-rule-name="our-nat-rule" dst-nat-rule-type="N/A" dst-nat-rule-name="N/A" protocol-id="6" policy-name="default-permit" source-zone-name="trust" destination-zone-name="untrust" session-id-32="3851" packets-from-client="161" bytes-from-client="9530" packets-from-server="96" bytes-from-server="9670" elapsed-time="23755" application="UNKNOWN" nested-application="UNKNOWN" username="N/A" roles="N/A" packet-incoming-interface="ge-0/0/1.0" encrypted="UNKNOWN" application-category="N/A" application-sub-category="N/A" application-risk="1" application-characteristics="N/A" secure-web-proxy-session-type="NA" peer-session-id="0" peer-source-address="0.0.0.0" peer-source-port="0" peer-destination-address="0.0.0.0" peer-destination-port="0" hostname="NA NA" src-vrf-grp="N/A" dst-vrf-grp="N/A"]
 <14>1 2020-07-13T16:12:05.530Z SRX100HM RT_FLOW - RT_FLOW_SESSION_CREATE [[email protected] source-address="10.1.1.100" source-port="49583" destination-address="175.16.199.1" destination-port="53" connection-tag="0" service-name="junos-dns-udp" nat-source-address="172.19.34.100" nat-source-port="30838" nat-destination-address="175.16.199.1" nat-destination-port="53" nat-connection-tag="0" src-nat-rule-type="source rule" src-nat-rule-name="our-nat-rule" dst-nat-rule-type="N/A" dst-nat-rule-name="N/A" protocol-id="17" policy-name="default-permit" source-zone-name="trust" destination-zone-name="untrust" session-id-32="15399" username="N/A" roles="N/A" packet-incoming-interface="ge-0/0/1.0" application="UNKNOWN" nested-application="UNKNOWN" encrypted="UNKNOWN" application-category="N/A" application-sub-category="N/A" application-risk="1" application-characteristics="N/A" src-vrf-grp="N/A" dst-vrf-grp="N/A"]
 <14>1 2020-07-13T16:12:05.530Z SRX100HM RT_FLOW - APPTRACK_SESSION_CLOSE [[email protected] reason="Closed by junos-alg" source-address="10.1.1.100" source-port="63381" destination-address="175.16.199.1" destination-port="53" service-name="junos-dns-udp" application="UNKNOWN" nested-application="UNKNOWN" nat-source-address="172.19.34.100" nat-source-port="26764" nat-destination-address="175.16.199.1" nat-destination-port="53" src-nat-rule-name="our-nat-rule" dst-nat-rule-name="N/A" protocol-id="17" policy-name="default-permit" source-zone-name="trust" destination-zone-name="untrust" session-id-32="15361" packets-from-client="1" bytes-from-client="66" packets-from-server="1" bytes-from-server="82" elapsed-time="3" username="N/A" roles="N/A" encrypted="No" profile-name="N/A" rule-name="N/A" routing-instance="default" destination-interface-name="ge-0/0/0.0" uplink-incoming-interface-name="N/A" uplink-tx-bytes="0" uplink-rx-bytes="0" category="N/A" sub-category="N/A" apbr-policy-name="N/A" multipath-rule-name="N/A" src-vrf-grp="N/A" dst-vrf-grp="N/A"]
-<14>1 2023-08-08T14:28:00.778-05:00 Route1- RT_FLOW - RT_FLOW_SESSION_DENY [source-address="192.168.1.1" source-port="39017" destination-address="8.8.4.4" destination-port="53" connection-tag="0" service-name="junos-dns-udp" protocol-id="17" icmp-type="0" policy-name="dns_deny_outbound" source-zone-name="trust" destination-zone-name="untrust" application="UNKNOWN" nested-application="UNKNOWN" username="N/A" roles="N/A" packet-incoming-interface="reth0.0" encrypted="No" reason="Denied by policy" session-id="85905209174" application-category="N/A" application-sub-category="N/A" application-risk="-1" application-characteristics="N/A" src-vrf-grp="N/A" dst-vrf-grp="N/A"]
+<14>1 2023-08-08T14:28:00.778-05:00 Route1- RT_FLOW - RT_FLOW_SESSION_DENY [[email protected] source-address="192.168.1.1" source-port="39017" destination-address="8.8.4.4" destination-port="53" connection-tag="0" service-name="junos-dns-udp" protocol-id="17" icmp-type="0" policy-name="dns_deny_outbound" source-zone-name="trust" destination-zone-name="untrust" application="UNKNOWN" nested-application="UNKNOWN" username="N/A" roles="N/A" packet-incoming-interface="reth0.0" encrypted="No" reason="Denied by policy" session-id="85905209174" application-category="N/A" application-sub-category="N/A" application-risk="-1" application-characteristics="N/A" src-vrf-grp="N/A" dst-vrf-grp="N/A"]

tests still pass, with expected event as shown above.

[novaksam]

@efd6 it appears that the GROK pattern used in 8.8.2 differs from the fixes you've applied to 8.9.

https://github.com/elastic/beats/blob/95be378c22fa51d8088f4c91751c6a8be4ccaf6d/x-pack/filebeat/module/juniper/srx/ingest/pipeline.yml#L8C1-L8C226

#36308

I suppose that means it's technically fixed? I have to stick to 8.8 for reasons but at least I know how to fix it.

[efd6]

Yes, this is fixed; no more changes will be applied to the 8.8 branch. I would suggest using the more general pattern that exists since that first element is not an email address, and I could find no docs to describe its format hence the "this isn't a key/value element"-pattern.