Palo Alto Module does not provide log.source.address

[strawgate]

For confirmed bugs, please report:

• Version: 7.5.0
• Operating System: Any
• Steps to Reproduce:

1. Setup a Filebeat with a Palo Alto Module
2. Forward logs from the Palo Alto device to the Filebeat
3. Notice that there is no log.source.address populated on the messages parsed through filebeat

[faec]

I'm not sure log.source.address is supposed to appear on panw messages... see the module's list of exported fields. Is panw.panos.source.nat.ip / panw.panos.source.nat.port what you're looking for?

[strawgate]

nat.ip and nat.port are related to the traffic going through the palo alto.

The palo alto is forwarding Syslog and so i'd expect that log.source.address would be populated similar to any other syslog source. The list of exported fields does not include standard fields -- for instance the cisco exported fields does not cover log.source.address: https://www.elastic.co/guide/en/beats/filebeat/master/exported-fields-cisco.html but those fields are present on the forwarded log messages

[faec]

Ah, no you're right, I just double checked and that field should be present. Can you share the filebeat.yml you're seeing this with?

[strawgate]

#####################
#
#=========================== Filebeat inputs =============================
filebeat.inputs:
   
#============================= Filebeat modules ===============================
filebeat.config.modules:
  path: ${path.config}/configs/panw/module.yml
  reload.enabled: false
output.logstash:
  hosts: ["172.18.0.1:4060"]
logging.metrics.period: 600
logging.json: true

and

# Module: panw
# Docs: https://www.elastic.co/guide/en/beats/filebeat/7.3/filebeat-module-panw.html
- module: panw
  panos:
    enabled: true
    var.syslog_host: 0.0.0.0
    var.syslog_port: 5060

[faec]

Ah, ok logstash could be a factor, but first, can you share an example of an event ingested with this configuration? I'd especially like to see whether the events are otherwise successfully parsed by the syslog input (this would be signaled by a "syslog" key with relevant metadata inside the indexed event), or if the input itself is having trouble.

[strawgate]

Here's the message as it comes from filebeat:

{
  "@timestamp": "2020-01-02T20:35:54.589Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "_doc",
    "version": "7.5.0",
    "truncated": false,
    "pipeline": "filebeat-7.5.0-panw-panos-pipeline"
  },
  "network": {
    "application": "ssl",
    "transport": "tcp",
    "community_id": "1:7WQBEq/QCPNFLId7r93vN98nPHQ="
  },
  "url": {
    "original": "consent.cmp.oath.com/"
  },
  "log": {
    "level": "informational"
  },
  "service": {
    "type": "panw"
  },
  "fileset": {
    "name": "panos"
  },
  "client": {
    "ip": "192.168.15.224",
    "port": "52993"
  },
  "panw": {
    "panos": {
      "source": {
        "zone": "trust",
        "interface": "ethernet1/2",
        "nat": {
          "ip": "192.168.1.63",
          "port": 10473
        }
      },
      "destination": {
        "nat": {
          "ip": "152.195.55.192",
          "port": 443
        },
        "zone": "untrust",
        "interface": "ethernet1/1"
      },
      "flow_id": "28143",
      "threat": {
        "resource": "consent.cmp.oath.com/",
        "name": "(9999)"
      },
      "url": {
        "category": "business-and-economy"
      },
      "network": {
        "pcap_id": "0",
        "nat": {
          "community_id": "1:nmxmtIja0z/MV5rgbBnScsKtW0U="
        }
      },
      "ruleset": "new_outbound_from_trust"
    }
  },
  "message": "Jan 02 14:39:35 PA-220 1,2018/11/30 16:44:36,012801096514,THREAT,url,2049,2018/11/30 16:44:36,192.168.15.224,152.195.55.192,192.168.1.63,152.195.55.192,new_outbound_from_trust,,,ssl,vsys1,trust,untrust,ethernet1/2,ethernet1/1,send_to_mac,2018/11/30 16:44:36,28143,1,52993,443,10473,443,0x403000,tcp,block-url,\"consent.cmp.oath.com/\",(9999),business-and-economy,informational,client-to-server,7737,0x2000000000000000,192.168.0.0-192.168.255.255,United States,0,,0,,,0,,,,,,,,0,0,0,0,0,,PA-220,,,,,0,,0,,N/A,unknown,AppThreat-0-0,0x0,0,4294967295",
  "input": {
    "type": "syslog"
  },
  "server": {
    "ip": "152.195.55.192",
    "port": "443"
  },
  "tags": [
    "pan-os"
  ],
  "event": {
    "timezone": "+00:00",
    "created": "2018/11/30 16:44:36",
    "outcome": "block-url",
    "module": "panw",
    "dataset": "panw.panos"
  },
  "ecs": {
    "version": "1.1.0"
  },
  "agent": {
    "id": "337e86b0-3526-44f5-a40c-333ed0dc73a1",
    "version": "7.5.0",
    "type": "filebeat",
    "ephemeral_id": "aa7933c4-0821-408e-aacf-dabd74c5fc0d",
    "hostname": "localhost.localdomain"
  },
  "host": {
    "name": "localhost.localdomain"
  },
  "observer": {
    "serial_number": "012801096514",
    "hostname": "PA-220"
  },
  "_temp_": {
    "message_type": "THREAT",
    "message_subtype": "url",
    "generated_time": "2018/11/30 16:44:36",
    "labels": "0x403000",
    "direction": "client-to-server"
  },
  "source": {
    "address": "192.168.15.224",
    "ip": "192.168.15.224",
    "nat": {
      "ip": "192.168.1.63",
      "port": "10473"
    },
    "port": "52993",
    "geo": {
      "country_iso_code": "192.168.0.0-192.168.255.255"
    }
  },
  "destination": {
    "geo": {
      "country_iso_code": "United States"
    },
    "ip": "152.195.55.192",
    "address": "152.195.55.192",
    "nat": {
      "ip": "152.195.55.192",
      "port": "443"
    },
    "port": "443"
  }
}

The panw tests also reflect this:
https://github.com/elastic/beats/blob/master/x-pack/filebeat/module/panw/panos/test/pan_inc_threat.log-expected.json
https://github.com/elastic/beats/blob/master/x-pack/filebeat/module/panw/panos/test/pan_inc_threat.log

[ycombinator]

I can indeed reproduce this bug locally with Filebeat 7.5.0. However, when I try to reproduce it with a Filebeat (x-pack/filebeat) built from the latest master, I'm unable to.

Performing a git bisect, it appears that this bug has recently been fixed by #15453. The bugfix is targeted for release in v7.6.0 and v7.5.2.