diff --git a/packages/gcp/_dev/build/docs/README.md b/packages/gcp/_dev/build/docs/README.md index e7cbc679e16..499c6e13408 100644 --- a/packages/gcp/_dev/build/docs/README.md +++ b/packages/gcp/_dev/build/docs/README.md @@ -207,3 +207,29 @@ The `dns` dataset collects queries that name servers resolve for your Virtual Pr {{fields "dns"}} {{event "dns"}} + +## Metrics + +### Billing + +The `billing` dataset collects GCP Billing information from Google Cloud BigQuery daily cost detail table. + +{{fields "billing"}} + +{{event "billing"}} + +### Compute + +The `compute` dataset is designed to fetch metrics for [Compute Engine](https://cloud.google.com/compute/) Virtual Machines in Google Cloud Platform. + +{{fields "compute"}} + +{{event "compute"}} + +### Firestore + +The `firestore` dataset fetches metrics from [Firestore](https://cloud.google.com/firestore/) in Google Cloud Platform. + +{{fields "firestore"}} + +{{event "firestore"}} \ No newline at end of file diff --git a/packages/gcp/_dev/build/docs/billing.md b/packages/gcp/_dev/build/docs/billing.md new file mode 100644 index 00000000000..f9cff298c3a --- /dev/null +++ b/packages/gcp/_dev/build/docs/billing.md @@ -0,0 +1,17 @@ +# Billing + +## Metrics + +The `billing` dataset collects [Cloud Billing Reports](https://cloud.google.com/billing/docs/reports) information from Google Cloud BigQuery daily cost detail table. BigQuery is a fully-managed, serverless data warehouse. Cloud Billing export to BigQuery enables you to export detailed Google Cloud billing data (such as usage, cost estimates, and pricing data) automatically throughout the day to a BigQuery dataset that you specify. Then you can access your Cloud Billing data from BigQuery for detailed analysis. + +Please see [export cloud billing data to BigQuery](https://cloud.google.com/billing/docs/how-to/export-data-bigquery) for more details on how to export billing data. + +In BigQuery dataset, detailed Google Cloud daily cost data is loaded into a data table named `gcp_billing_export_v1_`. There is a defined schema for Google Cloud daily cost data that is exported to BigQuery. Please see [daily cost detail data schema](https://cloud.google.com/billing/docs/how-to/export-data-bigquery-tables#data-schema) for more details. + +## Sample Event + +{{event "billing"}} + +## Exported fields + +{{fields "billing"}} \ No newline at end of file diff --git a/packages/gcp/_dev/build/docs/compute.md b/packages/gcp/_dev/build/docs/compute.md new file mode 100644 index 00000000000..e69eef3d2e4 --- /dev/null +++ b/packages/gcp/_dev/build/docs/compute.md @@ -0,0 +1,15 @@ +# Compute + +## Metrics + +The `compute` dataset is designed to fetch metrics for [Compute Engine](https://cloud.google.com/compute/) Virtual Machines in Google Cloud Platform. It contains all metrics exported from the [GCP Cloud Monitoring API](https://cloud.google.com/monitoring/api/metrics_gcp#gcp-compute). + +Extra labels and metadata are also extracted using the [Compute API](https://cloud.google.com/compute/docs/reference/rest/v1/instances/get). This is enough to get most of the info associated with a metric like Compute labels and metadata and metric specific Labels. + +## Sample Event + +{{event "compute"}} + +## Exported fields + +{{fields "compute"}} \ No newline at end of file diff --git a/packages/gcp/_dev/build/docs/firestore.md b/packages/gcp/_dev/build/docs/firestore.md new file mode 100644 index 00000000000..c4c371b3126 --- /dev/null +++ b/packages/gcp/_dev/build/docs/firestore.md @@ -0,0 +1,15 @@ +# Firestore + +## Metrics + +The `firestore` dataset fetches metrics from [Firestore](https://cloud.google.com/firestore/) in Google Cloud Platform. It contains all metrics exported from the [GCP Firestore Monitoring API](https://cloud.google.com/monitoring/api/metrics_gcp#gcp-firestore). + +You can specify a single region to fetch metrics like `us-central1`. Be aware that GCP Storage does not use zones so `us-central1-a` will return nothing. If no region is specified, it will return metrics from all buckets. + +## Sample Event + +{{event "firestore"}} + +## Exported fields + +{{fields "firestore"}} \ No newline at end of file diff --git a/packages/gcp/changelog.yml b/packages/gcp/changelog.yml index 942672f462c..14cbadfb9af 100644 --- a/packages/gcp/changelog.yml +++ b/packages/gcp/changelog.yml @@ -1,4 +1,27 @@ # newer versions go on top +- version: 2.0.0 + changes: + - description: | + Move configurations to support metrics. This change is breaking, as it moves + some configuration from the top level variables to data stream variables. + + This change involves `project_id`, `credentials_file` and `credentials_json` + variables that are moved from input level configuration to package level + configuration (as those variables are reused across all inputs/data streams). + + Users with GCP integration enabled will need to input values for these + variables again when upgrading the policies to this version. + type: breaking-change + link: https://github.com/elastic/integrations/pull/2707 + - description: Add GCP Billing Data Stream + type: enhancement + link: https://github.com/elastic/integrations/pull/2141 + - description: Add GCP Compute Data Stream + type: enhancement + link: https://github.com/elastic/integrations/pull/2301 + - description: Add GCP Firestore Data stream + type: enhancement + link: https://github.com/elastic/integrations/pull/2704 - version: "1.10.0" changes: - description: Update package to ECS 8.3.0. diff --git a/packages/gcp/data_stream/audit/_dev/test/system/test-pubsub-config.yml b/packages/gcp/data_stream/audit/_dev/test/system/test-pubsub-config.yml index af58f65a919..2cabbe0984c 100644 --- a/packages/gcp/data_stream/audit/_dev/test/system/test-pubsub-config.yml +++ b/packages/gcp/data_stream/audit/_dev/test/system/test-pubsub-config.yml @@ -1,7 +1,6 @@ service: gcppubsub-emulator input: gcp-pubsub vars: - alternative_host: "{{Hostname}}:{{Port}}" credentials_json: |- { "type": "service_account", @@ -18,5 +17,6 @@ vars: project_id: audit data_stream: vars: + alternative_host: "{{Hostname}}:{{Port}}" subscription_name: subscription topic: topic diff --git a/packages/gcp/data_stream/audit/manifest.yml b/packages/gcp/data_stream/audit/manifest.yml index 0c5919ce10f..9386ed5ea01 100644 --- a/packages/gcp/data_stream/audit/manifest.yml +++ b/packages/gcp/data_stream/audit/manifest.yml @@ -27,6 +27,13 @@ streams: required: true show_user: false default: false + - name: alternative_host + type: text + title: Alternative host + multi: false + required: false + show_user: false + description: "Overrides the default Pub/Sub service address and disables TLS. For testing." - name: tags type: text title: Tags diff --git a/packages/gcp/data_stream/billing/agent/stream/stream.yml.hbs b/packages/gcp/data_stream/billing/agent/stream/stream.yml.hbs new file mode 100644 index 00000000000..ed6242e5bab --- /dev/null +++ b/packages/gcp/data_stream/billing/agent/stream/stream.yml.hbs @@ -0,0 +1,12 @@ +metricsets: ["billing"] +period: {{period}} +project_id: {{project_id}} +{{#if credentials_file}} +credentials_file_path: {{credentials_file}} +{{/if}} +{{#if credentials_json}} +credentials_json: '{{credentials_json}}' +{{/if}} +dataset_id: {{dataset_id}} +table_pattern: {{table_pattern}} +cost_type: {{cost_type}} diff --git a/packages/gcp/data_stream/billing/fields/agent.yml b/packages/gcp/data_stream/billing/fields/agent.yml new file mode 100644 index 00000000000..da4e652c53b --- /dev/null +++ b/packages/gcp/data_stream/billing/fields/agent.yml @@ -0,0 +1,198 @@ +- name: cloud + title: Cloud + group: 2 + description: Fields related to the cloud or infrastructure the events are coming from. + footnote: 'Examples: If Metricbeat is running on an EC2 host and fetches data from its host, the cloud info contains the data about this machine. If Metricbeat runs on a remote machine outside the cloud and fetches data from a service running in the cloud, the field contains cloud data from the machine the service is running on.' + type: group + fields: + - name: account.id + level: extended + type: keyword + ignore_above: 1024 + description: 'The cloud account or organization id used to identify different entities in a multi-tenant environment. + + Examples: AWS account id, Google Cloud ORG Id, or other unique identifier.' + example: 666777888999 + - name: availability_zone + level: extended + type: keyword + ignore_above: 1024 + description: Availability zone in which this host is running. + example: us-east-1c + - name: instance.id + level: extended + type: keyword + ignore_above: 1024 + description: Instance ID of the host machine. + example: i-1234567890abcdef0 + - name: instance.name + level: extended + type: keyword + ignore_above: 1024 + description: Instance name of the host machine. + - name: machine.type + level: extended + type: keyword + ignore_above: 1024 + description: Machine type of the host machine. + example: t2.medium + - name: provider + level: extended + type: keyword + ignore_above: 1024 + description: Name of the cloud provider. Example values are aws, azure, gcp, or digitalocean. + example: aws + - name: region + level: extended + type: keyword + ignore_above: 1024 + description: Region in which this host is running. + example: us-east-1 + - name: project.id + type: keyword + description: Name of the project in Google Cloud. + - name: image.id + type: keyword + description: Image ID for the cloud instance. +- name: container + title: Container + group: 2 + description: 'Container fields are used for meta information about the specific container that is the source of information. + + These fields help correlate data based containers from any runtime.' + type: group + fields: + - name: id + level: core + type: keyword + ignore_above: 1024 + description: Unique container id. + - name: image.name + level: extended + type: keyword + ignore_above: 1024 + description: Name of the image the container was built on. + - name: labels + level: extended + type: object + object_type: keyword + description: Image labels. + - name: name + level: extended + type: keyword + ignore_above: 1024 + description: Container name. +- name: host + title: Host + group: 2 + description: 'A host is defined as a general computing instance. + + ECS host.* fields should be populated with details about the host on which the event happened, or from which the measurement was taken. Host types include hardware, virtual machines, Docker containers, and Kubernetes nodes.' + type: group + fields: + - name: architecture + level: core + type: keyword + ignore_above: 1024 + description: Operating system architecture. + example: x86_64 + - name: domain + level: extended + type: keyword + ignore_above: 1024 + description: 'Name of the domain of which the host is a member. + + For example, on Windows this could be the host''s Active Directory domain or NetBIOS domain name. For Linux this could be the domain of the host''s LDAP provider.' + example: CONTOSO + default_field: false + - name: hostname + level: core + type: keyword + ignore_above: 1024 + description: 'Hostname of the host. + + It normally contains what the `hostname` command returns on the host machine.' + - name: id + level: core + type: keyword + ignore_above: 1024 + description: 'Unique host id. + + As hostname is not always unique, use values that are meaningful in your environment. + + Example: The current usage of `beat.name`.' + - name: ip + level: core + type: ip + description: Host ip addresses. + - name: mac + level: core + type: keyword + ignore_above: 1024 + description: Host mac addresses. + - name: name + level: core + type: keyword + ignore_above: 1024 + description: 'Name of the host. + + It can contain what `hostname` returns on Unix systems, the fully qualified domain name, or a name specified by the user. The sender decides which value to use.' + - name: os.family + level: extended + type: keyword + ignore_above: 1024 + description: OS family (such as redhat, debian, freebsd, windows). + example: debian + - name: os.kernel + level: extended + type: keyword + ignore_above: 1024 + description: Operating system kernel version as a raw string. + example: 4.4.0-112-generic + - name: os.name + level: extended + type: keyword + ignore_above: 1024 + multi_fields: + - name: text + type: text + norms: false + default_field: false + description: Operating system name, without the version. + example: Mac OS X + - name: os.platform + level: extended + type: keyword + ignore_above: 1024 + description: Operating system platform (such centos, ubuntu, windows). + example: darwin + - name: os.version + level: extended + type: keyword + ignore_above: 1024 + description: Operating system version as a raw string. + example: 10.14.1 + - name: type + level: core + type: keyword + ignore_above: 1024 + description: 'Type of host. + + For Cloud providers this can be the machine type like `t2.medium`. If vm, this could be the container, for example, or other information meaningful in your environment.' + - name: containerized + type: boolean + description: > + If the host is a container. + + - name: os.build + type: keyword + example: "18D109" + description: > + OS build information. + + - name: os.codename + type: keyword + example: "stretch" + description: > + OS codename, if any. + diff --git a/packages/gcp/data_stream/billing/fields/base-fields.yml b/packages/gcp/data_stream/billing/fields/base-fields.yml new file mode 100644 index 00000000000..c6cc7150750 --- /dev/null +++ b/packages/gcp/data_stream/billing/fields/base-fields.yml @@ -0,0 +1,20 @@ +- name: data_stream.type + type: constant_keyword + description: Data stream type. +- name: data_stream.dataset + type: constant_keyword + description: Data stream dataset. +- name: data_stream.namespace + type: constant_keyword + description: Data stream namespace. +- name: '@timestamp' + type: date + description: Event timestamp. +- name: event.module + type: constant_keyword + description: Event module + value: gcp +- name: event.dataset + type: constant_keyword + description: Event dataset + value: gcp.billing diff --git a/packages/gcp/data_stream/billing/fields/ecs.yml b/packages/gcp/data_stream/billing/fields/ecs.yml new file mode 100644 index 00000000000..83e3f6f1225 --- /dev/null +++ b/packages/gcp/data_stream/billing/fields/ecs.yml @@ -0,0 +1,24 @@ +- external: ecs + name: cloud +- external: ecs + name: cloud.account.id +- external: ecs + name: cloud.account.name +- external: ecs + name: cloud.availability_zone +- external: ecs + name: cloud.instance.id +- external: ecs + name: cloud.machine.type +- external: ecs + name: cloud.provider +- external: ecs + name: cloud.region +- external: ecs + name: ecs.version +- external: ecs + name: error +- external: ecs + name: error.message +- external: ecs + name: service.type diff --git a/packages/gcp/data_stream/billing/fields/fields.yml b/packages/gcp/data_stream/billing/fields/fields.yml new file mode 100644 index 00000000000..01a7e615bf3 --- /dev/null +++ b/packages/gcp/data_stream/billing/fields/fields.yml @@ -0,0 +1,22 @@ +- name: gcp.billing + type: group + description: Google Cloud Billing metrics + fields: + - name: cost_type + type: keyword + description: Cost types include regular, tax, adjustment, and rounding_error. + - name: invoice_month + type: keyword + description: Billing report month. + - name: project_id + type: keyword + description: Project ID of the billing report belongs to. + - name: project_name + type: keyword + description: Project Name of the billing report belongs to. + - name: total + type: float + description: Total billing amount. + - name: billing_account_id + type: keyword + description: Project Billing Account ID. diff --git a/packages/gcp/data_stream/billing/manifest.yml b/packages/gcp/data_stream/billing/manifest.yml new file mode 100644 index 00000000000..0b2342e9492 --- /dev/null +++ b/packages/gcp/data_stream/billing/manifest.yml @@ -0,0 +1,34 @@ +title: "GCP Billing Metrics" +type: metrics +streams: + - input: gcp/metrics + title: GCP Billing Metrics + description: Collect GCP Billing Metrics + vars: + - name: period + type: text + title: Period + default: 24h + - name: dataset_id + type: text + title: Dataset ID + multi: false + required: true + show_user: true + description: "Dataset ID that points to the top-level container which contains the actual billing tables." + - name: table_pattern + type: text + title: Table pattern + multi: false + required: true + show_user: true + description: "Daily cost detail billing table name prefix." + default: gcp_billing_export_v1 + - name: cost_type + type: text + title: Cost Type + multi: false + required: true + show_user: true + description: "The type of cost this line item represents: regular, tax, adjustment, or rounding error" + default: regular diff --git a/packages/gcp/data_stream/billing/sample_event.json b/packages/gcp/data_stream/billing/sample_event.json new file mode 100644 index 00000000000..2acd0b43085 --- /dev/null +++ b/packages/gcp/data_stream/billing/sample_event.json @@ -0,0 +1,35 @@ +{ + "@timestamp": "2017-10-12T08:05:34.853Z", + "cloud": { + "account": { + "id": "01475F-5B1080-1137E7" + }, + "project": { + "id": "elastic-bi", + "name": "elastic-containerlib-prod" + }, + "provider": "gcp" + }, + "event": { + "dataset": "gcp.billing", + "duration": 115000, + "module": "gcp" + }, + "gcp": { + "billing": { + "billing_account_id": "01475F-5B1080-1137E7", + "cost_type": "regular", + "invoice_month": "202106", + "project_id": "containerlib-prod-12763", + "project_name": "elastic-containerlib-prod", + "total": 4717.170681 + } + }, + "metricset": { + "name": "billing", + "period": 10000 + }, + "service": { + "type": "gcp" + } +} \ No newline at end of file diff --git a/packages/gcp/data_stream/compute/agent/stream/stream.yml.hbs b/packages/gcp/data_stream/compute/agent/stream/stream.yml.hbs new file mode 100644 index 00000000000..7876288f953 --- /dev/null +++ b/packages/gcp/data_stream/compute/agent/stream/stream.yml.hbs @@ -0,0 +1,16 @@ +metricsets: ["compute"] +period: {{period}} +project_id: {{project_id}} +{{#if credentials_file}} +credentials_file_path: {{credentials_file}} +{{/if}} +{{#if credentials_json}} +credentials_json: '{{credentials_json}}' +{{/if}} +{{#if region}} +region: {{region}} +{{/if}} +{{#if zone}} +zone: {{zone}} +{{/if}} +exclude_labels: {{exclude_labels}} diff --git a/packages/gcp/data_stream/compute/fields/agent.yml b/packages/gcp/data_stream/compute/fields/agent.yml new file mode 100644 index 00000000000..2a31d79f494 --- /dev/null +++ b/packages/gcp/data_stream/compute/fields/agent.yml @@ -0,0 +1,198 @@ +- name: cloud + title: Cloud + group: 2 + description: Fields related to the cloud or infrastructure the events are coming from. + footnote: 'Examples: If Metricbeat is running on an GCP Compute VM and fetches data from its host, the cloud info contains the data about this machine. If Metricbeat runs on a remote machine outside the cloud and fetches data from a service running in the cloud, the field contains cloud data from the machine the service is running on.' + type: group + fields: + - name: account.id + level: extended + type: keyword + ignore_above: 1024 + description: 'The cloud account or organization id used to identify different entities in a multi-tenant environment. + + Examples: AWS account id, Google Cloud ORG Id, or other unique identifier.' + example: 666777888999 + - name: availability_zone + level: extended + type: keyword + ignore_above: 1024 + description: Availability zone in which this host is running. + example: us-east-1c + - name: instance.id + level: extended + type: keyword + ignore_above: 1024 + description: Instance ID of the host machine. + example: i-1234567890abcdef0 + - name: instance.name + level: extended + type: keyword + ignore_above: 1024 + description: Instance name of the host machine. + - name: machine.type + level: extended + type: keyword + ignore_above: 1024 + description: Machine type of the host machine. + example: t2.medium + - name: provider + level: extended + type: keyword + ignore_above: 1024 + description: Name of the cloud provider. Example values are aws, azure, gcp, or digitalocean. + example: aws + - name: region + level: extended + type: keyword + ignore_above: 1024 + description: Region in which this host is running. + example: us-east-1 + - name: project.id + type: keyword + description: Name of the project in Google Cloud. + - name: image.id + type: keyword + description: Image ID for the cloud instance. +- name: container + title: Container + group: 2 + description: 'Container fields are used for meta information about the specific container that is the source of information. + + These fields help correlate data based containers from any runtime.' + type: group + fields: + - name: id + level: core + type: keyword + ignore_above: 1024 + description: Unique container id. + - name: image.name + level: extended + type: keyword + ignore_above: 1024 + description: Name of the image the container was built on. + - name: labels + level: extended + type: object + object_type: keyword + description: Image labels. + - name: name + level: extended + type: keyword + ignore_above: 1024 + description: Container name. +- name: host + title: Host + group: 2 + description: 'A host is defined as a general computing instance. + + ECS host.* fields should be populated with details about the host on which the event happened, or from which the measurement was taken. Host types include hardware, virtual machines, Docker containers, and Kubernetes nodes.' + type: group + fields: + - name: architecture + level: core + type: keyword + ignore_above: 1024 + description: Operating system architecture. + example: x86_64 + - name: domain + level: extended + type: keyword + ignore_above: 1024 + description: 'Name of the domain of which the host is a member. + + For example, on Windows this could be the host''s Active Directory domain or NetBIOS domain name. For Linux this could be the domain of the host''s LDAP provider.' + example: CONTOSO + default_field: false + - name: hostname + level: core + type: keyword + ignore_above: 1024 + description: 'Hostname of the host. + + It normally contains what the `hostname` command returns on the host machine.' + - name: id + level: core + type: keyword + ignore_above: 1024 + description: 'Unique host id. + + As hostname is not always unique, use values that are meaningful in your environment. + + Example: The current usage of `beat.name`.' + - name: ip + level: core + type: ip + description: Host ip addresses. + - name: mac + level: core + type: keyword + ignore_above: 1024 + description: Host mac addresses. + - name: name + level: core + type: keyword + ignore_above: 1024 + description: 'Name of the host. + + It can contain what `hostname` returns on Unix systems, the fully qualified domain name, or a name specified by the user. The sender decides which value to use.' + - name: os.family + level: extended + type: keyword + ignore_above: 1024 + description: OS family (such as redhat, debian, freebsd, windows). + example: debian + - name: os.kernel + level: extended + type: keyword + ignore_above: 1024 + description: Operating system kernel version as a raw string. + example: 4.4.0-112-generic + - name: os.name + level: extended + type: keyword + ignore_above: 1024 + multi_fields: + - name: text + type: text + norms: false + default_field: false + description: Operating system name, without the version. + example: Mac OS X + - name: os.platform + level: extended + type: keyword + ignore_above: 1024 + description: Operating system platform (such centos, ubuntu, windows). + example: darwin + - name: os.version + level: extended + type: keyword + ignore_above: 1024 + description: Operating system version as a raw string. + example: 10.14.1 + - name: type + level: core + type: keyword + ignore_above: 1024 + description: 'Type of host. + + For Cloud providers this can be the machine type like `t2.medium`. If vm, this could be the container, for example, or other information meaningful in your environment.' + - name: containerized + type: boolean + description: > + If the host is a container. + + - name: os.build + type: keyword + example: "18D109" + description: > + OS build information. + + - name: os.codename + type: keyword + example: "stretch" + description: > + OS codename, if any. + diff --git a/packages/gcp/data_stream/compute/fields/base-fields.yml b/packages/gcp/data_stream/compute/fields/base-fields.yml new file mode 100644 index 00000000000..e53dc6c3e2a --- /dev/null +++ b/packages/gcp/data_stream/compute/fields/base-fields.yml @@ -0,0 +1,20 @@ +- name: data_stream.type + type: constant_keyword + description: Data stream type. +- name: data_stream.dataset + type: constant_keyword + description: Data stream dataset. +- name: data_stream.namespace + type: constant_keyword + description: Data stream namespace. +- name: '@timestamp' + type: date + description: Event timestamp. +- name: event.module + type: constant_keyword + description: Event module + value: gcp +- name: event.dataset + type: constant_keyword + description: Event dataset + value: gcp.compute diff --git a/packages/gcp/data_stream/compute/fields/ecs.yml b/packages/gcp/data_stream/compute/fields/ecs.yml new file mode 100644 index 00000000000..83e3f6f1225 --- /dev/null +++ b/packages/gcp/data_stream/compute/fields/ecs.yml @@ -0,0 +1,24 @@ +- external: ecs + name: cloud +- external: ecs + name: cloud.account.id +- external: ecs + name: cloud.account.name +- external: ecs + name: cloud.availability_zone +- external: ecs + name: cloud.instance.id +- external: ecs + name: cloud.machine.type +- external: ecs + name: cloud.provider +- external: ecs + name: cloud.region +- external: ecs + name: ecs.version +- external: ecs + name: error +- external: ecs + name: error.message +- external: ecs + name: service.type diff --git a/packages/gcp/data_stream/compute/fields/fields.yml b/packages/gcp/data_stream/compute/fields/fields.yml new file mode 100644 index 00000000000..c55330dd657 --- /dev/null +++ b/packages/gcp/data_stream/compute/fields/fields.yml @@ -0,0 +1,61 @@ +- name: gcp.compute + description: Google Cloud Compute metrics + type: group + fields: + - name: firewall.dropped.bytes + type: long + description: Incoming bytes dropped by the firewall + - name: firewall.dropped_packets_count.value + type: long + description: Incoming packets dropped by the firewall + - name: instance.cpu.reserved_cores.value + type: double + description: Number of cores reserved on the host of the instance + - name: instance.cpu.usage_time.sec + type: double + description: Usage for all cores in seconds + - name: instance.cpu.usage.pct + type: double + description: The fraction of the allocated CPU that is currently in use on the instance + - name: instance.disk.read.bytes + type: long + description: Count of bytes read from disk + - name: instance.disk.read_ops_count.value + type: long + description: Count of disk read IO operations + - name: instance.disk.write.bytes + type: long + description: Count of bytes written to disk + - name: instance.disk.write_ops_count.value + type: long + description: Count of disk write IO operations + - name: instance.memory.balloon.ram_size.value + type: long + description: The total amount of memory in the VM. This metric is only available for VMs that belong to the e2 family. + - name: instance.memory.balloon.ram_used.value + type: long + description: Memory currently used in the VM. This metric is only available for VMs that belong to the e2 family. + - name: instance.memory.balloon.swap_in.bytes + type: long + description: The amount of memory read into the guest from its own swap space. This metric is only available for VMs that belong to the e2 family. + - name: instance.memory.balloon.swap_out.bytes + type: long + description: The amount of memory written from the guest to its own swap space. This metric is only available for VMs that belong to the e2 family. + - name: instance.network.ingress.bytes + type: long + description: Count of bytes received from the network + - name: instance.network.ingress.packets.count + type: long + description: Count of packets received from the network + - name: instance.network.egress.bytes + type: long + description: Count of bytes sent over the network + - name: instance.network.egress.packets.count + type: long + description: Count of packets sent over the network + - name: instance.uptime.sec + type: long + description: Number of seconds the VM has been running. + - name: instance.uptime_total.sec + type: long + description: Elapsed time since the VM was started, in seconds. Sampled every 60 seconds. After sampling, data is not visible for up to 120 seconds. diff --git a/packages/gcp/data_stream/compute/fields/package-fields.yml b/packages/gcp/data_stream/compute/fields/package-fields.yml new file mode 100644 index 00000000000..d8ccb93f502 --- /dev/null +++ b/packages/gcp/data_stream/compute/fields/package-fields.yml @@ -0,0 +1,31 @@ +- name: gcp + description: >- + GCP module + fields: + - name: labels + type: object + description: >- + GCP monitoring metrics labels + fields: + - name: user.* + type: object + object_type: keyword + - name: metadata.* + type: object + object_type: keyword + - name: metrics.* + type: object + object_type: keyword + - name: system.* + type: object + object_type: keyword + - name: resource.* + type: object + object_type: keyword + - name: "metrics.*.*.*.*" + type: object + object_type: double + object_type_mapping_type: "*" + description: > + Metrics that returned from Google Cloud API query. + diff --git a/packages/gcp/data_stream/compute/manifest.yml b/packages/gcp/data_stream/compute/manifest.yml new file mode 100644 index 00000000000..240d96806f0 --- /dev/null +++ b/packages/gcp/data_stream/compute/manifest.yml @@ -0,0 +1,31 @@ +title: "GCP Compute Metrics" +type: metrics +streams: + - input: gcp/metrics + title: GCP Compute Metrics + description: Collect GCP Compute Metrics + vars: + - name: zone + type: text + title: GCP Zone + multi: false + required: false + show_user: true + - name: region + type: text + title: GCP Region + multi: false + required: false + show_user: true + - name: period + type: text + title: Period + default: 10s + required: true + - name: exclude_labels + type: bool + title: Exclude Labels + description: Exclude additional labels from metrics + multi: false + required: false + show_user: true diff --git a/packages/gcp/data_stream/compute/sample_event.json b/packages/gcp/data_stream/compute/sample_event.json new file mode 100644 index 00000000000..62aabe7bddd --- /dev/null +++ b/packages/gcp/data_stream/compute/sample_event.json @@ -0,0 +1,84 @@ +{ + "@timestamp": "2017-10-12T08:05:34.853Z", + "cloud": { + "account": { + "id": "elastic-obs-integrations-dev", + "name": "elastic-obs-integrations-dev" + }, + "instance": { + "id": "4751091017865185079", + "name": "gke-cluster-1-default-pool-6617a8aa-5clh" + }, + "machine": { + "type": "e2-medium" + }, + "provider": "gcp", + "availability_zone": "us-central1-c", + "region": "us-central1" + }, + "event": { + "dataset": "gcp.compute", + "duration": 115000, + "module": "gcp" + }, + "gcp": { + "compute": { + "firewall": { + "dropped": { + "bytes": 421 + }, + "dropped_packets_count": { + "value": 4 + } + }, + "instance": { + "cpu": { + "reserved_cores": { + "value": 1 + }, + "usage": { + "pct": 0.07259952346383708 + }, + "usage_time": { + "sec": 4.355971407830225 + } + }, + "memory": { + "balloon": { + "ram_size": { + "value": 4128378880 + }, + "ram_used": { + "value": 2190848000 + }, + "swap_in": { + "bytes": 0 + }, + "swap_out": { + "bytes": 0 + } + } + }, + "uptime": { + "sec": 60.00000000000091 + } + } + }, + "labels": { + "user": { + "goog-gke-node": "" + } + } + }, + "host": { + "id": "4751091017865185079", + "name": "gke-cluster-1-default-pool-6617a8aa-5clh" + }, + "metricset": { + "name": "compute", + "period": 10000 + }, + "service": { + "type": "gcp" + } +} \ No newline at end of file diff --git a/packages/gcp/data_stream/dns/_dev/test/system/test-pubsub-config.yml b/packages/gcp/data_stream/dns/_dev/test/system/test-pubsub-config.yml index 21a23f5c000..4322dbc19cc 100644 --- a/packages/gcp/data_stream/dns/_dev/test/system/test-pubsub-config.yml +++ b/packages/gcp/data_stream/dns/_dev/test/system/test-pubsub-config.yml @@ -1,12 +1,22 @@ service: gcppubsub-emulator input: gcp-pubsub vars: - alternative_host: "{{Hostname}}:{{Port}}" - credentials_json: > - {"foo": "bar"} - + credentials_json: | + { + "type": "service_account", + "project_id": "foo", + "private_key_id": "x", + "private_key": "", + "client_email": "foo@bar.com", + "client_id": "0", + "auth_uri": "https://accounts.google.com/o/oauth2/auth", + "token_uri": "https://oauth2.googleapis.com/token", + "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs", + "client_x509_cert_url": "https://foo.bar/path" + } project_id: dns data_stream: vars: + alternative_host: "{{Hostname}}:{{Port}}" subscription_name: subscription topic: topic diff --git a/packages/gcp/data_stream/dns/manifest.yml b/packages/gcp/data_stream/dns/manifest.yml index c8190efa2e3..202ad033f78 100644 --- a/packages/gcp/data_stream/dns/manifest.yml +++ b/packages/gcp/data_stream/dns/manifest.yml @@ -27,6 +27,13 @@ streams: required: true show_user: false default: false + - name: alternative_host + type: text + title: Alternative host + multi: false + required: false + show_user: false + description: "Overrides the default Pub/Sub service address and disables TLS. For testing." - name: tags type: text title: Tags diff --git a/packages/gcp/data_stream/firestore/agent/stream/stream.yml.hbs b/packages/gcp/data_stream/firestore/agent/stream/stream.yml.hbs new file mode 100644 index 00000000000..97f3875279c --- /dev/null +++ b/packages/gcp/data_stream/firestore/agent/stream/stream.yml.hbs @@ -0,0 +1,16 @@ +metricsets: ["firestore"] +period: {{period}} +project_id: {{project_id}} +{{#if credentials_file}} +credentials_file_path: {{credentials_file}} +{{/if}} +{{#if credentials_json}} +credentials_json: '{{credentials_json}}' +{{/if}} +{{#if region}} +region: {{region}} +{{/if}} +{{#if zone}} +zone: {{zone}} +{{/if}} +exclude_labels: {{exclude_labels}} diff --git a/packages/gcp/data_stream/firestore/fields/agent.yml b/packages/gcp/data_stream/firestore/fields/agent.yml new file mode 100644 index 00000000000..2a31d79f494 --- /dev/null +++ b/packages/gcp/data_stream/firestore/fields/agent.yml @@ -0,0 +1,198 @@ +- name: cloud + title: Cloud + group: 2 + description: Fields related to the cloud or infrastructure the events are coming from. + footnote: 'Examples: If Metricbeat is running on an GCP Compute VM and fetches data from its host, the cloud info contains the data about this machine. If Metricbeat runs on a remote machine outside the cloud and fetches data from a service running in the cloud, the field contains cloud data from the machine the service is running on.' + type: group + fields: + - name: account.id + level: extended + type: keyword + ignore_above: 1024 + description: 'The cloud account or organization id used to identify different entities in a multi-tenant environment. + + Examples: AWS account id, Google Cloud ORG Id, or other unique identifier.' + example: 666777888999 + - name: availability_zone + level: extended + type: keyword + ignore_above: 1024 + description: Availability zone in which this host is running. + example: us-east-1c + - name: instance.id + level: extended + type: keyword + ignore_above: 1024 + description: Instance ID of the host machine. + example: i-1234567890abcdef0 + - name: instance.name + level: extended + type: keyword + ignore_above: 1024 + description: Instance name of the host machine. + - name: machine.type + level: extended + type: keyword + ignore_above: 1024 + description: Machine type of the host machine. + example: t2.medium + - name: provider + level: extended + type: keyword + ignore_above: 1024 + description: Name of the cloud provider. Example values are aws, azure, gcp, or digitalocean. + example: aws + - name: region + level: extended + type: keyword + ignore_above: 1024 + description: Region in which this host is running. + example: us-east-1 + - name: project.id + type: keyword + description: Name of the project in Google Cloud. + - name: image.id + type: keyword + description: Image ID for the cloud instance. +- name: container + title: Container + group: 2 + description: 'Container fields are used for meta information about the specific container that is the source of information. + + These fields help correlate data based containers from any runtime.' + type: group + fields: + - name: id + level: core + type: keyword + ignore_above: 1024 + description: Unique container id. + - name: image.name + level: extended + type: keyword + ignore_above: 1024 + description: Name of the image the container was built on. + - name: labels + level: extended + type: object + object_type: keyword + description: Image labels. + - name: name + level: extended + type: keyword + ignore_above: 1024 + description: Container name. +- name: host + title: Host + group: 2 + description: 'A host is defined as a general computing instance. + + ECS host.* fields should be populated with details about the host on which the event happened, or from which the measurement was taken. Host types include hardware, virtual machines, Docker containers, and Kubernetes nodes.' + type: group + fields: + - name: architecture + level: core + type: keyword + ignore_above: 1024 + description: Operating system architecture. + example: x86_64 + - name: domain + level: extended + type: keyword + ignore_above: 1024 + description: 'Name of the domain of which the host is a member. + + For example, on Windows this could be the host''s Active Directory domain or NetBIOS domain name. For Linux this could be the domain of the host''s LDAP provider.' + example: CONTOSO + default_field: false + - name: hostname + level: core + type: keyword + ignore_above: 1024 + description: 'Hostname of the host. + + It normally contains what the `hostname` command returns on the host machine.' + - name: id + level: core + type: keyword + ignore_above: 1024 + description: 'Unique host id. + + As hostname is not always unique, use values that are meaningful in your environment. + + Example: The current usage of `beat.name`.' + - name: ip + level: core + type: ip + description: Host ip addresses. + - name: mac + level: core + type: keyword + ignore_above: 1024 + description: Host mac addresses. + - name: name + level: core + type: keyword + ignore_above: 1024 + description: 'Name of the host. + + It can contain what `hostname` returns on Unix systems, the fully qualified domain name, or a name specified by the user. The sender decides which value to use.' + - name: os.family + level: extended + type: keyword + ignore_above: 1024 + description: OS family (such as redhat, debian, freebsd, windows). + example: debian + - name: os.kernel + level: extended + type: keyword + ignore_above: 1024 + description: Operating system kernel version as a raw string. + example: 4.4.0-112-generic + - name: os.name + level: extended + type: keyword + ignore_above: 1024 + multi_fields: + - name: text + type: text + norms: false + default_field: false + description: Operating system name, without the version. + example: Mac OS X + - name: os.platform + level: extended + type: keyword + ignore_above: 1024 + description: Operating system platform (such centos, ubuntu, windows). + example: darwin + - name: os.version + level: extended + type: keyword + ignore_above: 1024 + description: Operating system version as a raw string. + example: 10.14.1 + - name: type + level: core + type: keyword + ignore_above: 1024 + description: 'Type of host. + + For Cloud providers this can be the machine type like `t2.medium`. If vm, this could be the container, for example, or other information meaningful in your environment.' + - name: containerized + type: boolean + description: > + If the host is a container. + + - name: os.build + type: keyword + example: "18D109" + description: > + OS build information. + + - name: os.codename + type: keyword + example: "stretch" + description: > + OS codename, if any. + diff --git a/packages/gcp/data_stream/firestore/fields/base-fields.yml b/packages/gcp/data_stream/firestore/fields/base-fields.yml new file mode 100644 index 00000000000..7d9cfc69efd --- /dev/null +++ b/packages/gcp/data_stream/firestore/fields/base-fields.yml @@ -0,0 +1,20 @@ +- name: data_stream.type + type: constant_keyword + description: Data stream type. +- name: data_stream.dataset + type: constant_keyword + description: Data stream dataset. +- name: data_stream.namespace + type: constant_keyword + description: Data stream namespace. +- name: '@timestamp' + type: date + description: Event timestamp. +- name: event.module + type: constant_keyword + description: Event module + value: gcp +- name: event.dataset + type: constant_keyword + description: Event dataset + value: gcp.firestore diff --git a/packages/gcp/data_stream/firestore/fields/ecs.yml b/packages/gcp/data_stream/firestore/fields/ecs.yml new file mode 100644 index 00000000000..83e3f6f1225 --- /dev/null +++ b/packages/gcp/data_stream/firestore/fields/ecs.yml @@ -0,0 +1,24 @@ +- external: ecs + name: cloud +- external: ecs + name: cloud.account.id +- external: ecs + name: cloud.account.name +- external: ecs + name: cloud.availability_zone +- external: ecs + name: cloud.instance.id +- external: ecs + name: cloud.machine.type +- external: ecs + name: cloud.provider +- external: ecs + name: cloud.region +- external: ecs + name: ecs.version +- external: ecs + name: error +- external: ecs + name: error.message +- external: ecs + name: service.type diff --git a/packages/gcp/data_stream/firestore/fields/fields.yml b/packages/gcp/data_stream/firestore/fields/fields.yml new file mode 100644 index 00000000000..e470f84b87a --- /dev/null +++ b/packages/gcp/data_stream/firestore/fields/fields.yml @@ -0,0 +1,13 @@ +- name: gcp.firestore + description: Google Cloud Firestore metrics + type: group + fields: + - name: document.delete.count + type: long + description: The number of successful document deletes. + - name: document.read.count + type: long + description: The number of successful document reads from queries or lookups. + - name: document.write.count + type: long + description: The number of successful document writes. diff --git a/packages/gcp/data_stream/firestore/fields/package-fields.yml b/packages/gcp/data_stream/firestore/fields/package-fields.yml new file mode 100644 index 00000000000..d8ccb93f502 --- /dev/null +++ b/packages/gcp/data_stream/firestore/fields/package-fields.yml @@ -0,0 +1,31 @@ +- name: gcp + description: >- + GCP module + fields: + - name: labels + type: object + description: >- + GCP monitoring metrics labels + fields: + - name: user.* + type: object + object_type: keyword + - name: metadata.* + type: object + object_type: keyword + - name: metrics.* + type: object + object_type: keyword + - name: system.* + type: object + object_type: keyword + - name: resource.* + type: object + object_type: keyword + - name: "metrics.*.*.*.*" + type: object + object_type: double + object_type_mapping_type: "*" + description: > + Metrics that returned from Google Cloud API query. + diff --git a/packages/gcp/data_stream/firestore/manifest.yml b/packages/gcp/data_stream/firestore/manifest.yml new file mode 100644 index 00000000000..0b4061f8a52 --- /dev/null +++ b/packages/gcp/data_stream/firestore/manifest.yml @@ -0,0 +1,31 @@ +title: "GCP Firestore Metrics" +type: metrics +streams: + - input: gcp/metrics + title: GCP Firestore Metrics + description: Collect GCP Firestore Metrics + vars: + - name: zone + type: text + title: GCP Zone + multi: false + required: false + show_user: true + - name: region + type: text + title: GCP Region + multi: false + required: false + show_user: true + - name: period + type: text + title: Period + default: 60s + required: true + - name: exclude_labels + type: bool + title: Exclude Labels + description: Exclude additional labels from metrics + multi: false + required: false + show_user: true diff --git a/packages/gcp/data_stream/firestore/sample_event.json b/packages/gcp/data_stream/firestore/sample_event.json new file mode 100644 index 00000000000..ddfe07c3a27 --- /dev/null +++ b/packages/gcp/data_stream/firestore/sample_event.json @@ -0,0 +1,55 @@ +{ + "@timestamp": "2017-10-12T08:05:34.853Z", + "cloud": { + "account": { + "id": "elastic-obs-integrations-dev", + "name": "elastic-obs-integrations-dev" + }, + "instance": { + "id": "4751091017865185079", + "name": "gke-cluster-1-default-pool-6617a8aa-5clh" + }, + "machine": { + "type": "e2-medium" + }, + "provider": "gcp", + "availability_zone": "us-central1-c", + "region": "us-central1" + }, + "event": { + "dataset": "gcp.firestore", + "duration": 115000, + "module": "gcp" + }, + "gcp": { + "firestore": { + "document": { + "delete": { + "count": 3 + }, + "read": { + "count": 10 + }, + "write": { + "count": 1 + } + } + }, + "labels": { + "user": { + "goog-gke-node": "" + } + } + }, + "host": { + "id": "4751091017865185079", + "name": "gke-cluster-1-default-pool-6617a8aa-5clh" + }, + "metricset": { + "name": "firestore", + "period": 10000 + }, + "service": { + "type": "gcp" + } +} \ No newline at end of file diff --git a/packages/gcp/data_stream/firewall/_dev/test/system/test-pubsub-config.yml b/packages/gcp/data_stream/firewall/_dev/test/system/test-pubsub-config.yml index 5978986e01c..b171a4364ba 100644 --- a/packages/gcp/data_stream/firewall/_dev/test/system/test-pubsub-config.yml +++ b/packages/gcp/data_stream/firewall/_dev/test/system/test-pubsub-config.yml @@ -1,12 +1,12 @@ service: gcppubsub-emulator input: gcp-pubsub vars: - alternative_host: "{{Hostname}}:{{Port}}" credentials_json: > {"foo": "bar"} project_id: firewall data_stream: vars: + alternative_host: "{{Hostname}}:{{Port}}" subscription_name: subscription topic: topic diff --git a/packages/gcp/data_stream/firewall/manifest.yml b/packages/gcp/data_stream/firewall/manifest.yml index 4d0f663192b..0bc29d382eb 100644 --- a/packages/gcp/data_stream/firewall/manifest.yml +++ b/packages/gcp/data_stream/firewall/manifest.yml @@ -27,6 +27,13 @@ streams: required: true show_user: false default: false + - name: alternative_host + type: text + title: Alternative host + multi: false + required: false + show_user: false + description: "Overrides the default Pub/Sub service address and disables TLS. For testing." - name: tags type: text title: Tags diff --git a/packages/gcp/data_stream/vpcflow/_dev/test/system/test-pubsub-config.yml b/packages/gcp/data_stream/vpcflow/_dev/test/system/test-pubsub-config.yml index 5dc8e1f0fa4..cff212ca960 100644 --- a/packages/gcp/data_stream/vpcflow/_dev/test/system/test-pubsub-config.yml +++ b/packages/gcp/data_stream/vpcflow/_dev/test/system/test-pubsub-config.yml @@ -1,11 +1,11 @@ service: gcppubsub-emulator input: gcp-pubsub vars: - alternative_host: "{{Hostname}}:{{Port}}" credentials_json: >- {"type":"service_account","project_id":"foo","private_key_id":"x","private_key":"","client_email":"foo@bar.com","client_id":"0","auth_uri":"https://accounts.google.com/o/oauth2/auth","token_uri":"https://oauth2.googleapis.com/token","auth_provider_x509_cert_url":"https://www.googleapis.com/oauth2/v1/certs","client_x509_cert_url":"https://foo.bar/path"} project_id: vpcflow data_stream: vars: + alternative_host: "{{Hostname}}:{{Port}}" subscription_name: subscription topic: topic diff --git a/packages/gcp/data_stream/vpcflow/manifest.yml b/packages/gcp/data_stream/vpcflow/manifest.yml index 61718d78235..d6fd91bbaba 100644 --- a/packages/gcp/data_stream/vpcflow/manifest.yml +++ b/packages/gcp/data_stream/vpcflow/manifest.yml @@ -27,6 +27,13 @@ streams: required: true show_user: false default: false + - name: alternative_host + type: text + title: Alternative host + multi: false + required: false + show_user: false + description: "Overrides the default Pub/Sub service address and disables TLS. For testing." - name: tags type: text title: Tags diff --git a/packages/gcp/docs/README.md b/packages/gcp/docs/README.md index 8a16c4568dc..81d2d895395 100644 --- a/packages/gcp/docs/README.md +++ b/packages/gcp/docs/README.md @@ -1155,3 +1155,390 @@ An example event for `dns` looks as following: ] } ``` + +## Metrics + +### Billing + +The `billing` dataset collects GCP Billing information from Google Cloud BigQuery daily cost detail table. + +**Exported fields** + +| Field | Description | Type | +|---|---|---| +| @timestamp | Event timestamp. | date | +| cloud | Fields related to the cloud or infrastructure the events are coming from. | group | +| cloud.account.id | The cloud account or organization id used to identify different entities in a multi-tenant environment. Examples: AWS account id, Google Cloud ORG Id, or other unique identifier. | keyword | +| cloud.account.name | The cloud account name or alias used to identify different entities in a multi-tenant environment. Examples: AWS account name, Google Cloud ORG display name. | keyword | +| cloud.availability_zone | Availability zone in which this host is running. | keyword | +| cloud.image.id | Image ID for the cloud instance. | keyword | +| cloud.instance.id | Instance ID of the host machine. | keyword | +| cloud.instance.name | Instance name of the host machine. | keyword | +| cloud.machine.type | Machine type of the host machine. | keyword | +| cloud.project.id | Name of the project in Google Cloud. | keyword | +| cloud.provider | Name of the cloud provider. Example values are aws, azure, gcp, or digitalocean. | keyword | +| cloud.region | Region in which this host is running. | keyword | +| container.id | Unique container id. | keyword | +| container.image.name | Name of the image the container was built on. | keyword | +| container.labels | Image labels. | object | +| container.name | Container name. | keyword | +| data_stream.dataset | Data stream dataset. | constant_keyword | +| data_stream.namespace | Data stream namespace. | constant_keyword | +| data_stream.type | Data stream type. | constant_keyword | +| ecs.version | ECS version this event conforms to. `ecs.version` is a required field and must exist in all events. When querying across multiple indices -- which may conform to slightly different ECS versions -- this field lets integrations adjust to the schema version of the events. | keyword | +| error | These fields can represent errors of any kind. Use them for errors that happen while fetching events or in cases where the event itself contains an error. | group | +| error.message | Error message. | match_only_text | +| event.dataset | Event dataset | constant_keyword | +| event.module | Event module | constant_keyword | +| gcp.billing.billing_account_id | Project Billing Account ID. | keyword | +| gcp.billing.cost_type | Cost types include regular, tax, adjustment, and rounding_error. | keyword | +| gcp.billing.invoice_month | Billing report month. | keyword | +| gcp.billing.project_id | Project ID of the billing report belongs to. | keyword | +| gcp.billing.project_name | Project Name of the billing report belongs to. | keyword | +| gcp.billing.total | Total billing amount. | float | +| host.architecture | Operating system architecture. | keyword | +| host.containerized | If the host is a container. | boolean | +| host.domain | Name of the domain of which the host is a member. For example, on Windows this could be the host's Active Directory domain or NetBIOS domain name. For Linux this could be the domain of the host's LDAP provider. | keyword | +| host.hostname | Hostname of the host. It normally contains what the `hostname` command returns on the host machine. | keyword | +| host.id | Unique host id. As hostname is not always unique, use values that are meaningful in your environment. Example: The current usage of `beat.name`. | keyword | +| host.ip | Host ip addresses. | ip | +| host.mac | Host mac addresses. | keyword | +| host.name | Name of the host. It can contain what `hostname` returns on Unix systems, the fully qualified domain name, or a name specified by the user. The sender decides which value to use. | keyword | +| host.os.build | OS build information. | keyword | +| host.os.codename | OS codename, if any. | keyword | +| host.os.family | OS family (such as redhat, debian, freebsd, windows). | keyword | +| host.os.kernel | Operating system kernel version as a raw string. | keyword | +| host.os.name | Operating system name, without the version. | keyword | +| host.os.name.text | Multi-field of `host.os.name`. | text | +| host.os.platform | Operating system platform (such centos, ubuntu, windows). | keyword | +| host.os.version | Operating system version as a raw string. | keyword | +| host.type | Type of host. For Cloud providers this can be the machine type like `t2.medium`. If vm, this could be the container, for example, or other information meaningful in your environment. | keyword | +| service.type | The type of the service data is collected from. The type can be used to group and correlate logs and metrics from one service type. Example: If logs or metrics are collected from Elasticsearch, `service.type` would be `elasticsearch`. | keyword | + + +An example event for `billing` looks as following: + +```json +{ + "@timestamp": "2017-10-12T08:05:34.853Z", + "cloud": { + "account": { + "id": "01475F-5B1080-1137E7" + }, + "project": { + "id": "elastic-bi", + "name": "elastic-containerlib-prod" + }, + "provider": "gcp" + }, + "event": { + "dataset": "gcp.billing", + "duration": 115000, + "module": "gcp" + }, + "gcp": { + "billing": { + "billing_account_id": "01475F-5B1080-1137E7", + "cost_type": "regular", + "invoice_month": "202106", + "project_id": "containerlib-prod-12763", + "project_name": "elastic-containerlib-prod", + "total": 4717.170681 + } + }, + "metricset": { + "name": "billing", + "period": 10000 + }, + "service": { + "type": "gcp" + } +} +``` + +### Compute + +The `compute` dataset is designed to fetch metrics for [Compute Engine](https://cloud.google.com/compute/) Virtual Machines in Google Cloud Platform. + +**Exported fields** + +| Field | Description | Type | +|---|---|---| +| @timestamp | Event timestamp. | date | +| cloud | Fields related to the cloud or infrastructure the events are coming from. | group | +| cloud.account.id | The cloud account or organization id used to identify different entities in a multi-tenant environment. Examples: AWS account id, Google Cloud ORG Id, or other unique identifier. | keyword | +| cloud.account.name | The cloud account name or alias used to identify different entities in a multi-tenant environment. Examples: AWS account name, Google Cloud ORG display name. | keyword | +| cloud.availability_zone | Availability zone in which this host is running. | keyword | +| cloud.image.id | Image ID for the cloud instance. | keyword | +| cloud.instance.id | Instance ID of the host machine. | keyword | +| cloud.instance.name | Instance name of the host machine. | keyword | +| cloud.machine.type | Machine type of the host machine. | keyword | +| cloud.project.id | Name of the project in Google Cloud. | keyword | +| cloud.provider | Name of the cloud provider. Example values are aws, azure, gcp, or digitalocean. | keyword | +| cloud.region | Region in which this host is running. | keyword | +| container.id | Unique container id. | keyword | +| container.image.name | Name of the image the container was built on. | keyword | +| container.labels | Image labels. | object | +| container.name | Container name. | keyword | +| data_stream.dataset | Data stream dataset. | constant_keyword | +| data_stream.namespace | Data stream namespace. | constant_keyword | +| data_stream.type | Data stream type. | constant_keyword | +| ecs.version | ECS version this event conforms to. `ecs.version` is a required field and must exist in all events. When querying across multiple indices -- which may conform to slightly different ECS versions -- this field lets integrations adjust to the schema version of the events. | keyword | +| error | These fields can represent errors of any kind. Use them for errors that happen while fetching events or in cases where the event itself contains an error. | group | +| error.message | Error message. | match_only_text | +| event.dataset | Event dataset | constant_keyword | +| event.module | Event module | constant_keyword | +| gcp.compute.firewall.dropped.bytes | Incoming bytes dropped by the firewall | long | +| gcp.compute.firewall.dropped_packets_count.value | Incoming packets dropped by the firewall | long | +| gcp.compute.instance.cpu.reserved_cores.value | Number of cores reserved on the host of the instance | double | +| gcp.compute.instance.cpu.usage.pct | The fraction of the allocated CPU that is currently in use on the instance | double | +| gcp.compute.instance.cpu.usage_time.sec | Usage for all cores in seconds | double | +| gcp.compute.instance.disk.read.bytes | Count of bytes read from disk | long | +| gcp.compute.instance.disk.read_ops_count.value | Count of disk read IO operations | long | +| gcp.compute.instance.disk.write.bytes | Count of bytes written to disk | long | +| gcp.compute.instance.disk.write_ops_count.value | Count of disk write IO operations | long | +| gcp.compute.instance.memory.balloon.ram_size.value | The total amount of memory in the VM. This metric is only available for VMs that belong to the e2 family. | long | +| gcp.compute.instance.memory.balloon.ram_used.value | Memory currently used in the VM. This metric is only available for VMs that belong to the e2 family. | long | +| gcp.compute.instance.memory.balloon.swap_in.bytes | The amount of memory read into the guest from its own swap space. This metric is only available for VMs that belong to the e2 family. | long | +| gcp.compute.instance.memory.balloon.swap_out.bytes | The amount of memory written from the guest to its own swap space. This metric is only available for VMs that belong to the e2 family. | long | +| gcp.compute.instance.network.egress.bytes | Count of bytes sent over the network | long | +| gcp.compute.instance.network.egress.packets.count | Count of packets sent over the network | long | +| gcp.compute.instance.network.ingress.bytes | Count of bytes received from the network | long | +| gcp.compute.instance.network.ingress.packets.count | Count of packets received from the network | long | +| gcp.compute.instance.uptime.sec | Number of seconds the VM has been running. | long | +| gcp.compute.instance.uptime_total.sec | Elapsed time since the VM was started, in seconds. Sampled every 60 seconds. After sampling, data is not visible for up to 120 seconds. | long | +| gcp.labels.metadata.\* | | object | +| gcp.labels.metrics.\* | | object | +| gcp.labels.resource.\* | | object | +| gcp.labels.system.\* | | object | +| gcp.labels.user.\* | | object | +| gcp.metrics.\*.\*.\*.\* | Metrics that returned from Google Cloud API query. | object | +| host.architecture | Operating system architecture. | keyword | +| host.containerized | If the host is a container. | boolean | +| host.domain | Name of the domain of which the host is a member. For example, on Windows this could be the host's Active Directory domain or NetBIOS domain name. For Linux this could be the domain of the host's LDAP provider. | keyword | +| host.hostname | Hostname of the host. It normally contains what the `hostname` command returns on the host machine. | keyword | +| host.id | Unique host id. As hostname is not always unique, use values that are meaningful in your environment. Example: The current usage of `beat.name`. | keyword | +| host.ip | Host ip addresses. | ip | +| host.mac | Host mac addresses. | keyword | +| host.name | Name of the host. It can contain what `hostname` returns on Unix systems, the fully qualified domain name, or a name specified by the user. The sender decides which value to use. | keyword | +| host.os.build | OS build information. | keyword | +| host.os.codename | OS codename, if any. | keyword | +| host.os.family | OS family (such as redhat, debian, freebsd, windows). | keyword | +| host.os.kernel | Operating system kernel version as a raw string. | keyword | +| host.os.name | Operating system name, without the version. | keyword | +| host.os.name.text | Multi-field of `host.os.name`. | text | +| host.os.platform | Operating system platform (such centos, ubuntu, windows). | keyword | +| host.os.version | Operating system version as a raw string. | keyword | +| host.type | Type of host. For Cloud providers this can be the machine type like `t2.medium`. If vm, this could be the container, for example, or other information meaningful in your environment. | keyword | +| service.type | The type of the service data is collected from. The type can be used to group and correlate logs and metrics from one service type. Example: If logs or metrics are collected from Elasticsearch, `service.type` would be `elasticsearch`. | keyword | + + +An example event for `compute` looks as following: + +```json +{ + "@timestamp": "2017-10-12T08:05:34.853Z", + "cloud": { + "account": { + "id": "elastic-obs-integrations-dev", + "name": "elastic-obs-integrations-dev" + }, + "instance": { + "id": "4751091017865185079", + "name": "gke-cluster-1-default-pool-6617a8aa-5clh" + }, + "machine": { + "type": "e2-medium" + }, + "provider": "gcp", + "availability_zone": "us-central1-c", + "region": "us-central1" + }, + "event": { + "dataset": "gcp.compute", + "duration": 115000, + "module": "gcp" + }, + "gcp": { + "compute": { + "firewall": { + "dropped": { + "bytes": 421 + }, + "dropped_packets_count": { + "value": 4 + } + }, + "instance": { + "cpu": { + "reserved_cores": { + "value": 1 + }, + "usage": { + "pct": 0.07259952346383708 + }, + "usage_time": { + "sec": 4.355971407830225 + } + }, + "memory": { + "balloon": { + "ram_size": { + "value": 4128378880 + }, + "ram_used": { + "value": 2190848000 + }, + "swap_in": { + "bytes": 0 + }, + "swap_out": { + "bytes": 0 + } + } + }, + "uptime": { + "sec": 60.00000000000091 + } + } + }, + "labels": { + "user": { + "goog-gke-node": "" + } + } + }, + "host": { + "id": "4751091017865185079", + "name": "gke-cluster-1-default-pool-6617a8aa-5clh" + }, + "metricset": { + "name": "compute", + "period": 10000 + }, + "service": { + "type": "gcp" + } +} +``` + +### Firestore + +The `firestore` dataset fetches metrics from [Firestore](https://cloud.google.com/firestore/) in Google Cloud Platform. + +**Exported fields** + +| Field | Description | Type | +|---|---|---| +| @timestamp | Event timestamp. | date | +| cloud | Fields related to the cloud or infrastructure the events are coming from. | group | +| cloud.account.id | The cloud account or organization id used to identify different entities in a multi-tenant environment. Examples: AWS account id, Google Cloud ORG Id, or other unique identifier. | keyword | +| cloud.account.name | The cloud account name or alias used to identify different entities in a multi-tenant environment. Examples: AWS account name, Google Cloud ORG display name. | keyword | +| cloud.availability_zone | Availability zone in which this host is running. | keyword | +| cloud.image.id | Image ID for the cloud instance. | keyword | +| cloud.instance.id | Instance ID of the host machine. | keyword | +| cloud.instance.name | Instance name of the host machine. | keyword | +| cloud.machine.type | Machine type of the host machine. | keyword | +| cloud.project.id | Name of the project in Google Cloud. | keyword | +| cloud.provider | Name of the cloud provider. Example values are aws, azure, gcp, or digitalocean. | keyword | +| cloud.region | Region in which this host, resource, or service is located. | keyword | +| container.id | Unique container id. | keyword | +| container.image.name | Name of the image the container was built on. | keyword | +| container.labels | Image labels. | object | +| container.name | Container name. | keyword | +| data_stream.dataset | Data stream dataset. | constant_keyword | +| data_stream.namespace | Data stream namespace. | constant_keyword | +| data_stream.type | Data stream type. | constant_keyword | +| ecs.version | ECS version this event conforms to. `ecs.version` is a required field and must exist in all events. When querying across multiple indices -- which may conform to slightly different ECS versions -- this field lets integrations adjust to the schema version of the events. | keyword | +| error | These fields can represent errors of any kind. Use them for errors that happen while fetching events or in cases where the event itself contains an error. | group | +| error.message | Error message. | match_only_text | +| event.dataset | Event dataset | constant_keyword | +| event.module | Event module | constant_keyword | +| gcp.firestore.document.delete.count | The number of successful document deletes. | long | +| gcp.firestore.document.read.count | The number of successful document reads from queries or lookups. | long | +| gcp.firestore.document.write.count | The number of successful document writes. | long | +| gcp.labels.metadata.\* | | object | +| gcp.labels.metrics.\* | | object | +| gcp.labels.resource.\* | | object | +| gcp.labels.system.\* | | object | +| gcp.labels.user.\* | | object | +| gcp.metrics.\*.\*.\*.\* | Metrics that returned from Google Cloud API query. | object | +| host.architecture | Operating system architecture. | keyword | +| host.containerized | If the host is a container. | boolean | +| host.domain | Name of the domain of which the host is a member. For example, on Windows this could be the host's Active Directory domain or NetBIOS domain name. For Linux this could be the domain of the host's LDAP provider. | keyword | +| host.hostname | Hostname of the host. It normally contains what the `hostname` command returns on the host machine. | keyword | +| host.id | Unique host id. As hostname is not always unique, use values that are meaningful in your environment. Example: The current usage of `beat.name`. | keyword | +| host.ip | Host ip addresses. | ip | +| host.mac | Host mac addresses. | keyword | +| host.name | Name of the host. It can contain what `hostname` returns on Unix systems, the fully qualified domain name, or a name specified by the user. The sender decides which value to use. | keyword | +| host.os.build | OS build information. | keyword | +| host.os.codename | OS codename, if any. | keyword | +| host.os.family | OS family (such as redhat, debian, freebsd, windows). | keyword | +| host.os.kernel | Operating system kernel version as a raw string. | keyword | +| host.os.name | Operating system name, without the version. | keyword | +| host.os.name.text | Multi-field of `host.os.name`. | text | +| host.os.platform | Operating system platform (such centos, ubuntu, windows). | keyword | +| host.os.version | Operating system version as a raw string. | keyword | +| host.type | Type of host. For Cloud providers this can be the machine type like `t2.medium`. If vm, this could be the container, for example, or other information meaningful in your environment. | keyword | +| service.type | The type of the service data is collected from. The type can be used to group and correlate logs and metrics from one service type. Example: If logs or metrics are collected from Elasticsearch, `service.type` would be `elasticsearch`. | keyword | + + +An example event for `firestore` looks as following: + +```json +{ + "@timestamp": "2017-10-12T08:05:34.853Z", + "cloud": { + "account": { + "id": "elastic-obs-integrations-dev", + "name": "elastic-obs-integrations-dev" + }, + "instance": { + "id": "4751091017865185079", + "name": "gke-cluster-1-default-pool-6617a8aa-5clh" + }, + "machine": { + "type": "e2-medium" + }, + "provider": "gcp", + "availability_zone": "us-central1-c", + "region": "us-central1" + }, + "event": { + "dataset": "gcp.firestore", + "duration": 115000, + "module": "gcp" + }, + "gcp": { + "firestore": { + "document": { + "delete": { + "count": 3 + }, + "read": { + "count": 10 + }, + "write": { + "count": 1 + } + } + }, + "labels": { + "user": { + "goog-gke-node": "" + } + } + }, + "host": { + "id": "4751091017865185079", + "name": "gke-cluster-1-default-pool-6617a8aa-5clh" + }, + "metricset": { + "name": "firestore", + "period": 10000 + }, + "service": { + "type": "gcp" + } +} +``` \ No newline at end of file diff --git a/packages/gcp/docs/billing.md b/packages/gcp/docs/billing.md new file mode 100644 index 00000000000..30701286feb --- /dev/null +++ b/packages/gcp/docs/billing.md @@ -0,0 +1,106 @@ +# Billing + +## Metrics + +The `billing` dataset collects [Cloud Billing Reports](https://cloud.google.com/billing/docs/reports) information from Google Cloud BigQuery daily cost detail table. BigQuery is a fully-managed, serverless data warehouse. Cloud Billing export to BigQuery enables you to export detailed Google Cloud billing data (such as usage, cost estimates, and pricing data) automatically throughout the day to a BigQuery dataset that you specify. Then you can access your Cloud Billing data from BigQuery for detailed analysis. + +Please see [export cloud billing data to BigQuery](https://cloud.google.com/billing/docs/how-to/export-data-bigquery) for more details on how to export billing data. + +In BigQuery dataset, detailed Google Cloud daily cost data is loaded into a data table named `gcp_billing_export_v1_`. There is a defined schema for Google Cloud daily cost data that is exported to BigQuery. Please see [daily cost detail data schema](https://cloud.google.com/billing/docs/how-to/export-data-bigquery-tables#data-schema) for more details. + +## Sample Event + +An example event for `billing` looks as following: + +```json +{ + "@timestamp": "2017-10-12T08:05:34.853Z", + "cloud": { + "account": { + "id": "01475F-5B1080-1137E7" + }, + "project": { + "id": "elastic-bi", + "name": "elastic-containerlib-prod" + }, + "provider": "gcp" + }, + "event": { + "dataset": "gcp.billing", + "duration": 115000, + "module": "gcp" + }, + "gcp": { + "billing": { + "billing_account_id": "01475F-5B1080-1137E7", + "cost_type": "regular", + "invoice_month": "202106", + "project_id": "containerlib-prod-12763", + "project_name": "elastic-containerlib-prod", + "total": 4717.170681 + } + }, + "metricset": { + "name": "billing", + "period": 10000 + }, + "service": { + "type": "gcp" + } +} +``` + +## Exported fields + +**Exported fields** + +| Field | Description | Type | +|---|---|---| +| @timestamp | Event timestamp. | date | +| cloud | Fields related to the cloud or infrastructure the events are coming from. | group | +| cloud.account.id | The cloud account or organization id used to identify different entities in a multi-tenant environment. Examples: AWS account id, Google Cloud ORG Id, or other unique identifier. | keyword | +| cloud.account.name | The cloud account name or alias used to identify different entities in a multi-tenant environment. Examples: AWS account name, Google Cloud ORG display name. | keyword | +| cloud.availability_zone | Availability zone in which this host is running. | keyword | +| cloud.image.id | Image ID for the cloud instance. | keyword | +| cloud.instance.id | Instance ID of the host machine. | keyword | +| cloud.instance.name | Instance name of the host machine. | keyword | +| cloud.machine.type | Machine type of the host machine. | keyword | +| cloud.project.id | Name of the project in Google Cloud. | keyword | +| cloud.provider | Name of the cloud provider. Example values are aws, azure, gcp, or digitalocean. | keyword | +| cloud.region | Region in which this host is running. | keyword | +| container.id | Unique container id. | keyword | +| container.image.name | Name of the image the container was built on. | keyword | +| container.labels | Image labels. | object | +| container.name | Container name. | keyword | +| data_stream.dataset | Data stream dataset. | constant_keyword | +| data_stream.namespace | Data stream namespace. | constant_keyword | +| data_stream.type | Data stream type. | constant_keyword | +| ecs.version | ECS version this event conforms to. `ecs.version` is a required field and must exist in all events. When querying across multiple indices -- which may conform to slightly different ECS versions -- this field lets integrations adjust to the schema version of the events. | keyword | +| error | These fields can represent errors of any kind. Use them for errors that happen while fetching events or in cases where the event itself contains an error. | group | +| error.message | Error message. | match_only_text | +| event.dataset | Event dataset | constant_keyword | +| event.module | Event module | constant_keyword | +| gcp.billing.billing_account_id | Project Billing Account ID. | keyword | +| gcp.billing.cost_type | Cost types include regular, tax, adjustment, and rounding_error. | keyword | +| gcp.billing.invoice_month | Billing report month. | keyword | +| gcp.billing.project_id | Project ID of the billing report belongs to. | keyword | +| gcp.billing.project_name | Project Name of the billing report belongs to. | keyword | +| gcp.billing.total | Total billing amount. | float | +| host.architecture | Operating system architecture. | keyword | +| host.containerized | If the host is a container. | boolean | +| host.domain | Name of the domain of which the host is a member. For example, on Windows this could be the host's Active Directory domain or NetBIOS domain name. For Linux this could be the domain of the host's LDAP provider. | keyword | +| host.hostname | Hostname of the host. It normally contains what the `hostname` command returns on the host machine. | keyword | +| host.id | Unique host id. As hostname is not always unique, use values that are meaningful in your environment. Example: The current usage of `beat.name`. | keyword | +| host.ip | Host ip addresses. | ip | +| host.mac | Host mac addresses. | keyword | +| host.name | Name of the host. It can contain what `hostname` returns on Unix systems, the fully qualified domain name, or a name specified by the user. The sender decides which value to use. | keyword | +| host.os.build | OS build information. | keyword | +| host.os.codename | OS codename, if any. | keyword | +| host.os.family | OS family (such as redhat, debian, freebsd, windows). | keyword | +| host.os.kernel | Operating system kernel version as a raw string. | keyword | +| host.os.name | Operating system name, without the version. | keyword | +| host.os.name.text | Multi-field of `host.os.name`. | text | +| host.os.platform | Operating system platform (such centos, ubuntu, windows). | keyword | +| host.os.version | Operating system version as a raw string. | keyword | +| host.type | Type of host. For Cloud providers this can be the machine type like `t2.medium`. If vm, this could be the container, for example, or other information meaningful in your environment. | keyword | +| service.type | The type of the service data is collected from. The type can be used to group and correlate logs and metrics from one service type. Example: If logs or metrics are collected from Elasticsearch, `service.type` would be `elasticsearch`. | keyword | diff --git a/packages/gcp/docs/compute.md b/packages/gcp/docs/compute.md new file mode 100644 index 00000000000..f44bacd9281 --- /dev/null +++ b/packages/gcp/docs/compute.md @@ -0,0 +1,172 @@ +# Compute + +## Metrics + +The `compute` dataset is designed to fetch metrics for [Compute Engine](https://cloud.google.com/compute/) Virtual Machines in Google Cloud Platform. It contains all metrics exported from the [GCP Cloud Monitoring API](https://cloud.google.com/monitoring/api/metrics_gcp#gcp-compute). + +Extra labels and metadata are also extracted using the [Compute API](https://cloud.google.com/compute/docs/reference/rest/v1/instances/get). This is enough to get most of the info associated with a metric like Compute labels and metadata and metric specific Labels. + +## Sample Event + +An example event for `compute` looks as following: + +```json +{ + "@timestamp": "2017-10-12T08:05:34.853Z", + "cloud": { + "account": { + "id": "elastic-obs-integrations-dev", + "name": "elastic-obs-integrations-dev" + }, + "instance": { + "id": "4751091017865185079", + "name": "gke-cluster-1-default-pool-6617a8aa-5clh" + }, + "machine": { + "type": "e2-medium" + }, + "provider": "gcp", + "availability_zone": "us-central1-c", + "region": "us-central1" + }, + "event": { + "dataset": "gcp.compute", + "duration": 115000, + "module": "gcp" + }, + "gcp": { + "compute": { + "firewall": { + "dropped": { + "bytes": 421 + }, + "dropped_packets_count": { + "value": 4 + } + }, + "instance": { + "cpu": { + "reserved_cores": { + "value": 1 + }, + "usage": { + "pct": 0.07259952346383708 + }, + "usage_time": { + "sec": 4.355971407830225 + } + }, + "memory": { + "balloon": { + "ram_size": { + "value": 4128378880 + }, + "ram_used": { + "value": 2190848000 + }, + "swap_in": { + "bytes": 0 + }, + "swap_out": { + "bytes": 0 + } + } + }, + "uptime": { + "sec": 60.00000000000091 + } + } + }, + "labels": { + "user": { + "goog-gke-node": "" + } + } + }, + "host": { + "id": "4751091017865185079", + "name": "gke-cluster-1-default-pool-6617a8aa-5clh" + }, + "metricset": { + "name": "compute", + "period": 10000 + }, + "service": { + "type": "gcp" + } +} +``` + +## Exported fields + +**Exported fields** + +| Field | Description | Type | +|---|---|---| +| @timestamp | Event timestamp. | date | +| cloud | Fields related to the cloud or infrastructure the events are coming from. | group | +| cloud.account.id | The cloud account or organization id used to identify different entities in a multi-tenant environment. Examples: AWS account id, Google Cloud ORG Id, or other unique identifier. | keyword | +| cloud.account.name | The cloud account name or alias used to identify different entities in a multi-tenant environment. Examples: AWS account name, Google Cloud ORG display name. | keyword | +| cloud.availability_zone | Availability zone in which this host is running. | keyword | +| cloud.image.id | Image ID for the cloud instance. | keyword | +| cloud.instance.id | Instance ID of the host machine. | keyword | +| cloud.instance.name | Instance name of the host machine. | keyword | +| cloud.machine.type | Machine type of the host machine. | keyword | +| cloud.project.id | Name of the project in Google Cloud. | keyword | +| cloud.provider | Name of the cloud provider. Example values are aws, azure, gcp, or digitalocean. | keyword | +| cloud.region | Region in which this host is running. | keyword | +| container.id | Unique container id. | keyword | +| container.image.name | Name of the image the container was built on. | keyword | +| container.labels | Image labels. | object | +| container.name | Container name. | keyword | +| data_stream.dataset | Data stream dataset. | constant_keyword | +| data_stream.namespace | Data stream namespace. | constant_keyword | +| data_stream.type | Data stream type. | constant_keyword | +| ecs.version | ECS version this event conforms to. `ecs.version` is a required field and must exist in all events. When querying across multiple indices -- which may conform to slightly different ECS versions -- this field lets integrations adjust to the schema version of the events. | keyword | +| error | These fields can represent errors of any kind. Use them for errors that happen while fetching events or in cases where the event itself contains an error. | group | +| error.message | Error message. | match_only_text | +| event.dataset | Event dataset | constant_keyword | +| event.module | Event module | constant_keyword | +| gcp.compute.firewall.dropped.bytes | Incoming bytes dropped by the firewall | long | +| gcp.compute.firewall.dropped_packets_count.value | Incoming packets dropped by the firewall | long | +| gcp.compute.instance.cpu.reserved_cores.value | Number of cores reserved on the host of the instance | double | +| gcp.compute.instance.cpu.usage.pct | The fraction of the allocated CPU that is currently in use on the instance | double | +| gcp.compute.instance.cpu.usage_time.sec | Usage for all cores in seconds | double | +| gcp.compute.instance.disk.read.bytes | Count of bytes read from disk | long | +| gcp.compute.instance.disk.read_ops_count.value | Count of disk read IO operations | long | +| gcp.compute.instance.disk.write.bytes | Count of bytes written to disk | long | +| gcp.compute.instance.disk.write_ops_count.value | Count of disk write IO operations | long | +| gcp.compute.instance.memory.balloon.ram_size.value | The total amount of memory in the VM. This metric is only available for VMs that belong to the e2 family. | long | +| gcp.compute.instance.memory.balloon.ram_used.value | Memory currently used in the VM. This metric is only available for VMs that belong to the e2 family. | long | +| gcp.compute.instance.memory.balloon.swap_in.bytes | The amount of memory read into the guest from its own swap space. This metric is only available for VMs that belong to the e2 family. | long | +| gcp.compute.instance.memory.balloon.swap_out.bytes | The amount of memory written from the guest to its own swap space. This metric is only available for VMs that belong to the e2 family. | long | +| gcp.compute.instance.network.egress.bytes | Count of bytes sent over the network | long | +| gcp.compute.instance.network.egress.packets.count | Count of packets sent over the network | long | +| gcp.compute.instance.network.ingress.bytes | Count of bytes received from the network | long | +| gcp.compute.instance.network.ingress.packets.count | Count of packets received from the network | long | +| gcp.compute.instance.uptime.sec | Number of seconds the VM has been running. | long | +| gcp.compute.instance.uptime_total.sec | Elapsed time since the VM was started, in seconds. Sampled every 60 seconds. After sampling, data is not visible for up to 120 seconds. | long | +| gcp.labels.metadata.\* | | object | +| gcp.labels.metrics.\* | | object | +| gcp.labels.resource.\* | | object | +| gcp.labels.system.\* | | object | +| gcp.labels.user.\* | | object | +| gcp.metrics.\*.\*.\*.\* | Metrics that returned from Google Cloud API query. | object | +| host.architecture | Operating system architecture. | keyword | +| host.containerized | If the host is a container. | boolean | +| host.domain | Name of the domain of which the host is a member. For example, on Windows this could be the host's Active Directory domain or NetBIOS domain name. For Linux this could be the domain of the host's LDAP provider. | keyword | +| host.hostname | Hostname of the host. It normally contains what the `hostname` command returns on the host machine. | keyword | +| host.id | Unique host id. As hostname is not always unique, use values that are meaningful in your environment. Example: The current usage of `beat.name`. | keyword | +| host.ip | Host ip addresses. | ip | +| host.mac | Host mac addresses. | keyword | +| host.name | Name of the host. It can contain what `hostname` returns on Unix systems, the fully qualified domain name, or a name specified by the user. The sender decides which value to use. | keyword | +| host.os.build | OS build information. | keyword | +| host.os.codename | OS codename, if any. | keyword | +| host.os.family | OS family (such as redhat, debian, freebsd, windows). | keyword | +| host.os.kernel | Operating system kernel version as a raw string. | keyword | +| host.os.name | Operating system name, without the version. | keyword | +| host.os.name.text | Multi-field of `host.os.name`. | text | +| host.os.platform | Operating system platform (such centos, ubuntu, windows). | keyword | +| host.os.version | Operating system version as a raw string. | keyword | +| host.type | Type of host. For Cloud providers this can be the machine type like `t2.medium`. If vm, this could be the container, for example, or other information meaningful in your environment. | keyword | +| service.type | The type of the service data is collected from. The type can be used to group and correlate logs and metrics from one service type. Example: If logs or metrics are collected from Elasticsearch, `service.type` would be `elasticsearch`. | keyword | diff --git a/packages/gcp/docs/firestore.md b/packages/gcp/docs/firestore.md new file mode 100644 index 00000000000..71627a47186 --- /dev/null +++ b/packages/gcp/docs/firestore.md @@ -0,0 +1,127 @@ +# Firestore + +## Metrics + +The `firestore` dataset fetches metrics from [Firestore](https://cloud.google.com/firestore/) in Google Cloud Platform. It contains all metrics exported from the [GCP Firestore Monitoring API](https://cloud.google.com/monitoring/api/metrics_gcp#gcp-firestore). + +You can specify a single region to fetch metrics like `us-central1`. Be aware that GCP Storage does not use zones so `us-central1-a` will return nothing. If no region is specified, it will return metrics from all buckets. + +## Sample Event + +An example event for `firestore` looks as following: + +```json +{ + "@timestamp": "2017-10-12T08:05:34.853Z", + "cloud": { + "account": { + "id": "elastic-obs-integrations-dev", + "name": "elastic-obs-integrations-dev" + }, + "instance": { + "id": "4751091017865185079", + "name": "gke-cluster-1-default-pool-6617a8aa-5clh" + }, + "machine": { + "type": "e2-medium" + }, + "provider": "gcp", + "availability_zone": "us-central1-c", + "region": "us-central1" + }, + "event": { + "dataset": "gcp.firestore", + "duration": 115000, + "module": "gcp" + }, + "gcp": { + "firestore": { + "document": { + "delete": { + "count": 3 + }, + "read": { + "count": 10 + }, + "write": { + "count": 1 + } + } + }, + "labels": { + "user": { + "goog-gke-node": "" + } + } + }, + "host": { + "id": "4751091017865185079", + "name": "gke-cluster-1-default-pool-6617a8aa-5clh" + }, + "metricset": { + "name": "firestore", + "period": 10000 + }, + "service": { + "type": "gcp" + } +} +``` + +## Exported fields + +**Exported fields** + +| Field | Description | Type | +|---|---|---| +| @timestamp | Event timestamp. | date | +| cloud | Fields related to the cloud or infrastructure the events are coming from. | group | +| cloud.account.id | The cloud account or organization id used to identify different entities in a multi-tenant environment. Examples: AWS account id, Google Cloud ORG Id, or other unique identifier. | keyword | +| cloud.account.name | The cloud account name or alias used to identify different entities in a multi-tenant environment. Examples: AWS account name, Google Cloud ORG display name. | keyword | +| cloud.availability_zone | Availability zone in which this host is running. | keyword | +| cloud.image.id | Image ID for the cloud instance. | keyword | +| cloud.instance.id | Instance ID of the host machine. | keyword | +| cloud.instance.name | Instance name of the host machine. | keyword | +| cloud.machine.type | Machine type of the host machine. | keyword | +| cloud.project.id | Name of the project in Google Cloud. | keyword | +| cloud.provider | Name of the cloud provider. Example values are aws, azure, gcp, or digitalocean. | keyword | +| cloud.region | Region in which this host, resource, or service is located. | keyword | +| container.id | Unique container id. | keyword | +| container.image.name | Name of the image the container was built on. | keyword | +| container.labels | Image labels. | object | +| container.name | Container name. | keyword | +| data_stream.dataset | Data stream dataset. | constant_keyword | +| data_stream.namespace | Data stream namespace. | constant_keyword | +| data_stream.type | Data stream type. | constant_keyword | +| ecs.version | ECS version this event conforms to. `ecs.version` is a required field and must exist in all events. When querying across multiple indices -- which may conform to slightly different ECS versions -- this field lets integrations adjust to the schema version of the events. | keyword | +| error | These fields can represent errors of any kind. Use them for errors that happen while fetching events or in cases where the event itself contains an error. | group | +| error.message | Error message. | match_only_text | +| event.dataset | Event dataset | constant_keyword | +| event.module | Event module | constant_keyword | +| gcp.firestore.document.delete.count | The number of successful document deletes. | long | +| gcp.firestore.document.read.count | The number of successful document reads from queries or lookups. | long | +| gcp.firestore.document.write.count | The number of successful document writes. | long | +| gcp.labels.metadata.\* | | object | +| gcp.labels.metrics.\* | | object | +| gcp.labels.resource.\* | | object | +| gcp.labels.system.\* | | object | +| gcp.labels.user.\* | | object | +| gcp.metrics.\*.\*.\*.\* | Metrics that returned from Google Cloud API query. | object | +| host.architecture | Operating system architecture. | keyword | +| host.containerized | If the host is a container. | boolean | +| host.domain | Name of the domain of which the host is a member. For example, on Windows this could be the host's Active Directory domain or NetBIOS domain name. For Linux this could be the domain of the host's LDAP provider. | keyword | +| host.hostname | Hostname of the host. It normally contains what the `hostname` command returns on the host machine. | keyword | +| host.id | Unique host id. As hostname is not always unique, use values that are meaningful in your environment. Example: The current usage of `beat.name`. | keyword | +| host.ip | Host ip addresses. | ip | +| host.mac | Host mac addresses. | keyword | +| host.name | Name of the host. It can contain what `hostname` returns on Unix systems, the fully qualified domain name, or a name specified by the user. The sender decides which value to use. | keyword | +| host.os.build | OS build information. | keyword | +| host.os.codename | OS codename, if any. | keyword | +| host.os.family | OS family (such as redhat, debian, freebsd, windows). | keyword | +| host.os.kernel | Operating system kernel version as a raw string. | keyword | +| host.os.name | Operating system name, without the version. | keyword | +| host.os.name.text | Multi-field of `host.os.name`. | text | +| host.os.platform | Operating system platform (such centos, ubuntu, windows). | keyword | +| host.os.version | Operating system version as a raw string. | keyword | +| host.type | Type of host. For Cloud providers this can be the machine type like `t2.medium`. If vm, this could be the container, for example, or other information meaningful in your environment. | keyword | +| service.type | The type of the service data is collected from. The type can be used to group and correlate logs and metrics from one service type. Example: If logs or metrics are collected from Elasticsearch, `service.type` would be `elasticsearch`. | keyword | diff --git a/packages/gcp/img/gcp-billing.png b/packages/gcp/img/gcp-billing.png new file mode 100644 index 00000000000..b697c285a11 Binary files /dev/null and b/packages/gcp/img/gcp-billing.png differ diff --git a/packages/gcp/img/gcp-compute.png b/packages/gcp/img/gcp-compute.png new file mode 100644 index 00000000000..d4d90d27ad3 Binary files /dev/null and b/packages/gcp/img/gcp-compute.png differ diff --git a/packages/gcp/kibana/dashboard/gcp-76c9e920-e890-11ea-bf8c-d13ebf358a78.json b/packages/gcp/kibana/dashboard/gcp-76c9e920-e890-11ea-bf8c-d13ebf358a78.json new file mode 100644 index 00000000000..d999e3b91f1 --- /dev/null +++ b/packages/gcp/kibana/dashboard/gcp-76c9e920-e890-11ea-bf8c-d13ebf358a78.json @@ -0,0 +1,164 @@ +{ + "attributes": { + "description": "Overview of Google Cloud Billing Metrics", + "hits": 0, + "kibanaSavedObjectMeta": { + "searchSourceJSON": { + "filter": [], + "query": { + "language": "kuery", + "query": "" + } + } + }, + "optionsJSON": { + "hidePanelTitles": false, + "useMargins": true + }, + "panelsJSON": [ + { + "embeddableConfig": { + "enhancements": {} + }, + "gridData": { + "h": 14, + "i": "2552123b-6ad6-4d63-89c3-0672ab428580", + "w": 8, + "x": 0, + "y": 0 + }, + "panelIndex": "2552123b-6ad6-4d63-89c3-0672ab428580", + "panelRefName": "panel_2552123b-6ad6-4d63-89c3-0672ab428580", + "title": "Filters", + "type": "visualization", + "version": "7.9.0" + }, + { + "embeddableConfig": { + "enhancements": {} + }, + "gridData": { + "h": 20, + "i": "2d3d3b79-0656-45c2-b051-4489484b625c", + "w": 10, + "x": 8, + "y": 0 + }, + "panelIndex": "2d3d3b79-0656-45c2-b051-4489484b625c", + "panelRefName": "panel_2d3d3b79-0656-45c2-b051-4489484b625c", + "title": "Cost Per Month", + "type": "lens", + "version": "7.9.0" + }, + { + "embeddableConfig": { + "enhancements": {} + }, + "gridData": { + "h": 20, + "i": "b737e597-cc4d-4437-859c-6d491679599d", + "w": 30, + "x": 18, + "y": 0 + }, + "panelIndex": "b737e597-cc4d-4437-859c-6d491679599d", + "panelRefName": "panel_b737e597-cc4d-4437-859c-6d491679599d", + "title": "Cost Per Project ID", + "type": "lens", + "version": "7.9.0" + }, + { + "embeddableConfig": { + "enhancements": {} + }, + "gridData": { + "h": 6, + "i": "9eedb0c7-2089-4e0f-af98-721034203aad", + "w": 8, + "x": 0, + "y": 14 + }, + "panelIndex": "9eedb0c7-2089-4e0f-af98-721034203aad", + "panelRefName": "panel_9eedb0c7-2089-4e0f-af98-721034203aad", + "title": "Total Number Of Projects", + "type": "lens", + "version": "7.9.0" + }, + { + "embeddableConfig": { + "enhancements": {} + }, + "gridData": { + "h": 15, + "i": "f4d0ebcb-ac15-4c31-ab57-7f22e0c3e02a", + "w": 48, + "x": 0, + "y": 20 + }, + "panelIndex": "f4d0ebcb-ac15-4c31-ab57-7f22e0c3e02a", + "panelRefName": "panel_f4d0ebcb-ac15-4c31-ab57-7f22e0c3e02a", + "title": "Cost Per Invoice Month", + "type": "lens", + "version": "7.9.0" + }, + { + "embeddableConfig": { + "enhancements": {} + }, + "gridData": { + "h": 15, + "i": "991e60a8-68eb-4c2b-ac9a-b553e90dd49d", + "w": 48, + "x": 0, + "y": 35 + }, + "panelIndex": "991e60a8-68eb-4c2b-ac9a-b553e90dd49d", + "panelRefName": "panel_991e60a8-68eb-4c2b-ac9a-b553e90dd49d", + "title": "Total Billing Cost", + "type": "lens", + "version": "7.9.0" + } + ], + "timeRestore": false, + "title": "[Metrics GCP] Billing Overview", + "version": 1 + }, + "coreMigrationVersion": "7.15.0", + "id": "gcp-76c9e920-e890-11ea-bf8c-d13ebf358a78", + "migrationVersion": { + "dashboard": "7.15.0" + }, + "references": [ + { + "id": "gcp-5d2f9160-e88e-11ea-bf8c-d13ebf358a78", + "name": "2552123b-6ad6-4d63-89c3-0672ab428580:panel_2552123b-6ad6-4d63-89c3-0672ab428580", + "type": "visualization" + }, + { + "id": "gcp-520c6f10-ec8a-11ea-a0ed-7fe6b565d158", + "name": "2d3d3b79-0656-45c2-b051-4489484b625c:panel_2d3d3b79-0656-45c2-b051-4489484b625c", + "type": "lens" + }, + { + "id": "gcp-057de170-e88d-11ea-bf8c-d13ebf358a78", + "name": "b737e597-cc4d-4437-859c-6d491679599d:panel_b737e597-cc4d-4437-859c-6d491679599d", + "type": "lens" + }, + { + "id": "gcp-dd835300-e88f-11ea-bf8c-d13ebf358a78", + "name": "9eedb0c7-2089-4e0f-af98-721034203aad:panel_9eedb0c7-2089-4e0f-af98-721034203aad", + "type": "lens" + }, + { + "id": "gcp-e6933020-e88d-11ea-bf8c-d13ebf358a78", + "name": "f4d0ebcb-ac15-4c31-ab57-7f22e0c3e02a:panel_f4d0ebcb-ac15-4c31-ab57-7f22e0c3e02a", + "type": "lens" + }, + { + "id": "gcp-73346db0-e88d-11ea-bf8c-d13ebf358a78", + "name": "991e60a8-68eb-4c2b-ac9a-b553e90dd49d:panel_991e60a8-68eb-4c2b-ac9a-b553e90dd49d", + "type": "lens" + } + ], + "type": "dashboard" +} \ No newline at end of file diff --git a/packages/gcp/kibana/dashboard/gcp-f40ee870-5e4a-11ea-a4f6-717338406083.json b/packages/gcp/kibana/dashboard/gcp-f40ee870-5e4a-11ea-a4f6-717338406083.json new file mode 100644 index 00000000000..c0c2d391340 --- /dev/null +++ b/packages/gcp/kibana/dashboard/gcp-f40ee870-5e4a-11ea-a4f6-717338406083.json @@ -0,0 +1,232 @@ +{ + "attributes": { + "description": "Overview of GCP Compute Metrics", + "hits": 0, + "kibanaSavedObjectMeta": { + "searchSourceJSON": { + "filter": [], + "query": { + "language": "kuery", + "query": "" + } + } + }, + "optionsJSON": { + "hidePanelTitles": false, + "useMargins": true + }, + "panelsJSON": [ + { + "embeddableConfig": { + "enhancements": {} + }, + "gridData": { + "h": 17, + "i": "28706ab2-1142-401d-9143-f4176a034c10", + "w": 7, + "x": 0, + "y": 0 + }, + "panelIndex": "28706ab2-1142-401d-9143-f4176a034c10", + "panelRefName": "panel_28706ab2-1142-401d-9143-f4176a034c10", + "title": "Filters", + "type": "visualization", + "version": "7.9.1" + }, + { + "embeddableConfig": { + "enhancements": {} + }, + "gridData": { + "h": 17, + "i": "2034fcc8-5cd7-4ee8-8c8f-99054f025b05", + "w": 10, + "x": 7, + "y": 0 + }, + "panelIndex": "2034fcc8-5cd7-4ee8-8c8f-99054f025b05", + "panelRefName": "panel_2034fcc8-5cd7-4ee8-8c8f-99054f025b05", + "title": "Instance Uptime", + "type": "visualization", + "version": "7.9.1" + }, + { + "embeddableConfig": { + "enhancements": {} + }, + "gridData": { + "h": 17, + "i": "5f6f2ecd-dcaf-4455-967c-ede6b38f431f", + "w": 31, + "x": 17, + "y": 0 + }, + "panelIndex": "5f6f2ecd-dcaf-4455-967c-ede6b38f431f", + "panelRefName": "panel_5f6f2ecd-dcaf-4455-967c-ede6b38f431f", + "title": "CPU Utilization", + "type": "visualization", + "version": "7.9.1" + }, + { + "embeddableConfig": { + "enhancements": {} + }, + "gridData": { + "h": 15, + "i": "9c6f36f5-c2b2-40f5-8ee3-af6131168842", + "w": 24, + "x": 0, + "y": 17 + }, + "panelIndex": "9c6f36f5-c2b2-40f5-8ee3-af6131168842", + "panelRefName": "panel_9c6f36f5-c2b2-40f5-8ee3-af6131168842", + "title": "Read I/O", + "type": "visualization", + "version": "7.9.1" + }, + { + "embeddableConfig": { + "enhancements": {} + }, + "gridData": { + "h": 15, + "i": "93906f63-42c9-4f30-9b2c-05041a9e1efe", + "w": 24, + "x": 24, + "y": 17 + }, + "panelIndex": "93906f63-42c9-4f30-9b2c-05041a9e1efe", + "panelRefName": "panel_93906f63-42c9-4f30-9b2c-05041a9e1efe", + "title": "Write I/O", + "type": "visualization", + "version": "7.9.1" + }, + { + "embeddableConfig": { + "enhancements": {} + }, + "gridData": { + "h": 15, + "i": "e1a4e862-dd00-409f-8746-8a8e4bc82807", + "w": 24, + "x": 0, + "y": 32 + }, + "panelIndex": "e1a4e862-dd00-409f-8746-8a8e4bc82807", + "panelRefName": "panel_e1a4e862-dd00-409f-8746-8a8e4bc82807", + "title": "Network Sent Bytes", + "type": "visualization", + "version": "7.9.1" + }, + { + "embeddableConfig": { + "enhancements": {} + }, + "gridData": { + "h": 15, + "i": "6f47ff85-3ec1-4f6f-a63b-1a56f0cfc9ce", + "w": 24, + "x": 24, + "y": 32 + }, + "panelIndex": "6f47ff85-3ec1-4f6f-a63b-1a56f0cfc9ce", + "panelRefName": "panel_6f47ff85-3ec1-4f6f-a63b-1a56f0cfc9ce", + "title": "Network Received Bytes", + "type": "visualization", + "version": "7.9.1" + }, + { + "embeddableConfig": { + "enhancements": {} + }, + "gridData": { + "h": 15, + "i": "00689e12-4cb3-49ad-ac33-dbe4279f446e", + "w": 24, + "x": 0, + "y": 47 + }, + "panelIndex": "00689e12-4cb3-49ad-ac33-dbe4279f446e", + "panelRefName": "panel_00689e12-4cb3-49ad-ac33-dbe4279f446e", + "title": "Firewall Dropped Bytes", + "type": "visualization", + "version": "7.9.1" + }, + { + "embeddableConfig": { + "enhancements": {} + }, + "gridData": { + "h": 15, + "i": "901e7bf5-35f5-4c1a-9627-27f6c20d2514", + "w": 24, + "x": 24, + "y": 47 + }, + "panelIndex": "901e7bf5-35f5-4c1a-9627-27f6c20d2514", + "panelRefName": "panel_901e7bf5-35f5-4c1a-9627-27f6c20d2514", + "title": "Firewall Dropped Packets", + "type": "visualization", + "version": "7.9.1" + } + ], + "timeRestore": false, + "title": "[Metrics GCP] Compute Overview", + "version": 1 + }, + "coreMigrationVersion": "8.0.0", + "id": "gcp-f40ee870-5e4a-11ea-a4f6-717338406083", + "migrationVersion": { + "dashboard": "7.14.0" + }, + "references": [ + { + "id": "gcp-3aa96470-5fc4-11ea-a4f6-717338406083", + "name": "28706ab2-1142-401d-9143-f4176a034c10:panel_28706ab2-1142-401d-9143-f4176a034c10", + "type": "visualization" + }, + { + "id": "gcp-da5bc460-63e1-11ea-b0ac-95d4ecb1fecd", + "name": "2034fcc8-5cd7-4ee8-8c8f-99054f025b05:panel_2034fcc8-5cd7-4ee8-8c8f-99054f025b05", + "type": "visualization" + }, + { + "id": "gcp-3f472ea0-5e47-11ea-a4f6-717338406083", + "name": "5f6f2ecd-dcaf-4455-967c-ede6b38f431f:panel_5f6f2ecd-dcaf-4455-967c-ede6b38f431f", + "type": "visualization" + }, + { + "id": "gcp-89513bc0-5e48-11ea-a4f6-717338406083", + "name": "9c6f36f5-c2b2-40f5-8ee3-af6131168842:panel_9c6f36f5-c2b2-40f5-8ee3-af6131168842", + "type": "visualization" + }, + { + "id": "gcp-95e1f050-5e48-11ea-a4f6-717338406083", + "name": "93906f63-42c9-4f30-9b2c-05041a9e1efe:panel_93906f63-42c9-4f30-9b2c-05041a9e1efe", + "type": "visualization" + }, + { + "id": "gcp-6f795e70-5e49-11ea-a4f6-717338406083", + "name": "e1a4e862-dd00-409f-8746-8a8e4bc82807:panel_e1a4e862-dd00-409f-8746-8a8e4bc82807", + "type": "visualization" + }, + { + "id": "gcp-43f45ba0-5e4a-11ea-a4f6-717338406083", + "name": "6f47ff85-3ec1-4f6f-a63b-1a56f0cfc9ce:panel_6f47ff85-3ec1-4f6f-a63b-1a56f0cfc9ce", + "type": "visualization" + }, + { + "id": "gcp-9d919d00-5e4d-11ea-a4f6-717338406083", + "name": "00689e12-4cb3-49ad-ac33-dbe4279f446e:panel_00689e12-4cb3-49ad-ac33-dbe4279f446e", + "type": "visualization" + }, + { + "id": "gcp-ef1508c0-5e4c-11ea-a4f6-717338406083", + "name": "901e7bf5-35f5-4c1a-9627-27f6c20d2514:panel_901e7bf5-35f5-4c1a-9627-27f6c20d2514", + "type": "visualization" + } + ], + "type": "dashboard", + "updated_at": "2021-08-04T16:31:09.611Z", + "version": "WzM3NzQsMV0=" +} \ No newline at end of file diff --git a/packages/gcp/kibana/lens/gcp-057de170-e88d-11ea-bf8c-d13ebf358a78.json b/packages/gcp/kibana/lens/gcp-057de170-e88d-11ea-bf8c-d13ebf358a78.json new file mode 100644 index 00000000000..a62be39b462 --- /dev/null +++ b/packages/gcp/kibana/lens/gcp-057de170-e88d-11ea-bf8c-d13ebf358a78.json @@ -0,0 +1,87 @@ +{ + "attributes": { + "state": { + "datasourceStates": { + "indexpattern": { + "layers": { + "4ca843af-63d7-46b9-a719-51a81eebf1f7": { + "columnOrder": [ + "2477291e-9021-4eb2-9fce-8da1ee792c49", + "10b91492-efef-490d-bc7a-c2074b2eae84" + ], + "columns": { + "10b91492-efef-490d-bc7a-c2074b2eae84": { + "dataType": "number", + "isBucketed": false, + "label": "Maximum of gcp.billing.total", + "operationType": "max", + "scale": "ratio", + "sourceField": "gcp.billing.total" + }, + "2477291e-9021-4eb2-9fce-8da1ee792c49": { + "customLabel": true, + "dataType": "string", + "isBucketed": true, + "label": "Cost Per Project ID", + "operationType": "terms", + "params": { + "orderBy": { + "columnId": "10b91492-efef-490d-bc7a-c2074b2eae84", + "type": "column" + }, + "orderDirection": "desc", + "size": 20 + }, + "scale": "ordinal", + "sourceField": "gcp.billing.project_id" + } + } + } + } + } + }, + "filters": [], + "query": { + "language": "kuery", + "query": "" + }, + "visualization": { + "layers": [ + { + "categoryDisplay": "default", + "groups": [ + "2477291e-9021-4eb2-9fce-8da1ee792c49" + ], + "layerId": "4ca843af-63d7-46b9-a719-51a81eebf1f7", + "layerType": "data", + "legendDisplay": "default", + "metric": "10b91492-efef-490d-bc7a-c2074b2eae84", + "nestedLegend": false, + "numberDisplay": "percent" + } + ], + "shape": "pie" + } + }, + "title": "Cost Per Project ID [Metrics GCP]", + "visualizationType": "lnsPie" + }, + "coreMigrationVersion": "7.15.0", + "id": "gcp-057de170-e88d-11ea-bf8c-d13ebf358a78", + "migrationVersion": { + "lens": "7.15.0" + }, + "references": [ + { + "id": "metrics-*", + "name": "indexpattern-datasource-current-indexpattern", + "type": "index-pattern" + }, + { + "id": "metrics-*", + "name": "indexpattern-datasource-layer-4ca843af-63d7-46b9-a719-51a81eebf1f7", + "type": "index-pattern" + } + ], + "type": "lens" +} \ No newline at end of file diff --git a/packages/gcp/kibana/lens/gcp-520c6f10-ec8a-11ea-a0ed-7fe6b565d158.json b/packages/gcp/kibana/lens/gcp-520c6f10-ec8a-11ea-a0ed-7fe6b565d158.json new file mode 100644 index 00000000000..6a75af55fbd --- /dev/null +++ b/packages/gcp/kibana/lens/gcp-520c6f10-ec8a-11ea-a0ed-7fe6b565d158.json @@ -0,0 +1,83 @@ +{ + "attributes": { + "state": { + "datasourceStates": { + "indexpattern": { + "layers": { + "e12171da-25a4-41ea-86d3-8fd71205c263": { + "columnOrder": [ + "6011e524-4646-410b-8d1c-06c281e8f7ed", + "f8ab301c-f139-4573-b233-ed8a3f717e24" + ], + "columns": { + "6011e524-4646-410b-8d1c-06c281e8f7ed": { + "customLabel": true, + "dataType": "string", + "isBucketed": true, + "label": "Invoice Month", + "operationType": "terms", + "params": { + "orderBy": { + "columnId": "f8ab301c-f139-4573-b233-ed8a3f717e24", + "type": "column" + }, + "orderDirection": "desc", + "size": 12 + }, + "scale": "ordinal", + "sourceField": "gcp.billing.invoice_month" + }, + "f8ab301c-f139-4573-b233-ed8a3f717e24": { + "customLabel": true, + "dataType": "number", + "isBucketed": false, + "label": "Total Billing Cost", + "operationType": "sum", + "scale": "ratio", + "sourceField": "gcp.billing.total" + } + } + } + } + } + }, + "filters": [], + "query": { + "language": "kuery", + "query": "" + }, + "visualization": { + "columns": [ + { + "columnId": "6011e524-4646-410b-8d1c-06c281e8f7ed" + }, + { + "columnId": "f8ab301c-f139-4573-b233-ed8a3f717e24" + } + ], + "layerId": "e12171da-25a4-41ea-86d3-8fd71205c263", + "layerType": "data" + } + }, + "title": "Total Cost Table [Metrics GCP]", + "visualizationType": "lnsDatatable" + }, + "coreMigrationVersion": "7.15.0", + "id": "gcp-520c6f10-ec8a-11ea-a0ed-7fe6b565d158", + "migrationVersion": { + "lens": "7.15.0" + }, + "references": [ + { + "id": "metrics-*", + "name": "indexpattern-datasource-current-indexpattern", + "type": "index-pattern" + }, + { + "id": "metrics-*", + "name": "indexpattern-datasource-layer-e12171da-25a4-41ea-86d3-8fd71205c263", + "type": "index-pattern" + } + ], + "type": "lens" +} \ No newline at end of file diff --git a/packages/gcp/kibana/lens/gcp-73346db0-e88d-11ea-bf8c-d13ebf358a78.json b/packages/gcp/kibana/lens/gcp-73346db0-e88d-11ea-bf8c-d13ebf358a78.json new file mode 100644 index 00000000000..98207850aba --- /dev/null +++ b/packages/gcp/kibana/lens/gcp-73346db0-e88d-11ea-bf8c-d13ebf358a78.json @@ -0,0 +1,153 @@ +{ + "attributes": { + "state": { + "datasourceStates": { + "indexpattern": { + "layers": { + "325e60ce-0fbd-42b0-82f6-b10df31fef6c": { + "columnOrder": [ + "faaaaf23-f362-4a00-be9e-8a155208a39e", + "c4bc659c-3e7c-41f2-bc38-32d9edee95e8", + "3041fc1b-ceb8-4188-b55d-d354819f267e" + ], + "columns": { + "3041fc1b-ceb8-4188-b55d-d354819f267e": { + "customLabel": true, + "dataType": "number", + "isBucketed": false, + "label": "Total Billing", + "operationType": "max", + "scale": "ratio", + "sourceField": "gcp.billing.total" + }, + "c4bc659c-3e7c-41f2-bc38-32d9edee95e8": { + "dataType": "date", + "isBucketed": true, + "label": "@timestamp", + "operationType": "date_histogram", + "params": { + "interval": "1d" + }, + "scale": "interval", + "sourceField": "@timestamp" + }, + "faaaaf23-f362-4a00-be9e-8a155208a39e": { + "customLabel": true, + "dataType": "string", + "isBucketed": true, + "label": "Project ID", + "operationType": "terms", + "params": { + "orderBy": { + "columnId": "3041fc1b-ceb8-4188-b55d-d354819f267e", + "type": "column" + }, + "orderDirection": "desc", + "size": 10 + }, + "scale": "ordinal", + "sourceField": "gcp.billing.project_id" + } + } + }, + "4ca843af-63d7-46b9-a719-51a81eebf1f7": { + "columnOrder": [ + "1164563d-d2b3-4067-bc7b-d694179182ed", + "10b91492-efef-490d-bc7a-c2074b2eae84" + ], + "columns": { + "10b91492-efef-490d-bc7a-c2074b2eae84": { + "customLabel": true, + "dataType": "number", + "isBucketed": false, + "label": "Total Billing Cost", + "operationType": "sum", + "scale": "ratio", + "sourceField": "gcp.billing.total" + }, + "1164563d-d2b3-4067-bc7b-d694179182ed": { + "dataType": "date", + "isBucketed": true, + "label": "@timestamp", + "operationType": "date_histogram", + "params": { + "interval": "1d" + }, + "scale": "interval", + "sourceField": "@timestamp" + }, + "e25f49de-f161-4be8-a8fc-519188a7776c": { + "customLabel": true, + "dataType": "string", + "isBucketed": true, + "label": "Cost", + "operationType": "terms", + "params": { + "orderBy": { + "columnId": "10b91492-efef-490d-bc7a-c2074b2eae84", + "type": "column" + }, + "orderDirection": "desc", + "size": 15 + }, + "scale": "ordinal", + "sourceField": "gcp.billing.project_id" + } + } + } + } + } + }, + "filters": [], + "query": { + "language": "kuery", + "query": "" + }, + "visualization": { + "fittingFunction": "None", + "layers": [ + { + "accessors": [ + "3041fc1b-ceb8-4188-b55d-d354819f267e" + ], + "layerId": "325e60ce-0fbd-42b0-82f6-b10df31fef6c", + "layerType": "data", + "seriesType": "bar_stacked", + "splitAccessor": "faaaaf23-f362-4a00-be9e-8a155208a39e", + "xAccessor": "c4bc659c-3e7c-41f2-bc38-32d9edee95e8" + } + ], + "legend": { + "isVisible": true, + "position": "right" + }, + "preferredSeriesType": "bar_stacked" + } + }, + "title": "Total Cost Bar Chart [Metrics GCP]", + "visualizationType": "lnsXY" + }, + "coreMigrationVersion": "7.15.0", + "id": "gcp-73346db0-e88d-11ea-bf8c-d13ebf358a78", + "migrationVersion": { + "lens": "7.15.0" + }, + "references": [ + { + "id": "metrics-*", + "name": "indexpattern-datasource-current-indexpattern", + "type": "index-pattern" + }, + { + "id": "metrics-*", + "name": "indexpattern-datasource-layer-325e60ce-0fbd-42b0-82f6-b10df31fef6c", + "type": "index-pattern" + }, + { + "id": "metrics-*", + "name": "indexpattern-datasource-layer-4ca843af-63d7-46b9-a719-51a81eebf1f7", + "type": "index-pattern" + } + ], + "type": "lens" +} \ No newline at end of file diff --git a/packages/gcp/kibana/lens/gcp-dd835300-e88f-11ea-bf8c-d13ebf358a78.json b/packages/gcp/kibana/lens/gcp-dd835300-e88f-11ea-bf8c-d13ebf358a78.json new file mode 100644 index 00000000000..92147debf47 --- /dev/null +++ b/packages/gcp/kibana/lens/gcp-dd835300-e88f-11ea-bf8c-d13ebf358a78.json @@ -0,0 +1,58 @@ +{ + "attributes": { + "state": { + "datasourceStates": { + "indexpattern": { + "layers": { + "4cb00ce3-c62e-46f3-90ce-b69c876b9605": { + "columnOrder": [ + "2f66b924-5392-4e5e-93fe-5b23a87068c1" + ], + "columns": { + "2f66b924-5392-4e5e-93fe-5b23a87068c1": { + "customLabel": true, + "dataType": "number", + "isBucketed": false, + "label": "", + "operationType": "unique_count", + "scale": "ratio", + "sourceField": "gcp.billing.project_id" + } + } + } + } + } + }, + "filters": [], + "query": { + "language": "kuery", + "query": "" + }, + "visualization": { + "accessor": "2f66b924-5392-4e5e-93fe-5b23a87068c1", + "layerId": "4cb00ce3-c62e-46f3-90ce-b69c876b9605", + "layerType": "data" + } + }, + "title": "Total Number Of Projects [Metrics GCP]", + "visualizationType": "lnsMetric" + }, + "coreMigrationVersion": "7.15.0", + "id": "gcp-dd835300-e88f-11ea-bf8c-d13ebf358a78", + "migrationVersion": { + "lens": "7.15.0" + }, + "references": [ + { + "id": "metrics-*", + "name": "indexpattern-datasource-current-indexpattern", + "type": "index-pattern" + }, + { + "id": "metrics-*", + "name": "indexpattern-datasource-layer-4cb00ce3-c62e-46f3-90ce-b69c876b9605", + "type": "index-pattern" + } + ], + "type": "lens" +} \ No newline at end of file diff --git a/packages/gcp/kibana/lens/gcp-e6933020-e88d-11ea-bf8c-d13ebf358a78.json b/packages/gcp/kibana/lens/gcp-e6933020-e88d-11ea-bf8c-d13ebf358a78.json new file mode 100644 index 00000000000..502ed7d0f7d --- /dev/null +++ b/packages/gcp/kibana/lens/gcp-e6933020-e88d-11ea-bf8c-d13ebf358a78.json @@ -0,0 +1,108 @@ +{ + "attributes": { + "state": { + "datasourceStates": { + "indexpattern": { + "layers": { + "4ca843af-63d7-46b9-a719-51a81eebf1f7": { + "columnOrder": [ + "e25f49de-f161-4be8-a8fc-519188a7776c", + "b92edf5e-58bc-4382-9cd5-19db2c332c93", + "af747bf6-66e9-4760-bbd8-3dae9c97159d" + ], + "columns": { + "af747bf6-66e9-4760-bbd8-3dae9c97159d": { + "customLabel": true, + "dataType": "number", + "isBucketed": false, + "label": "Total Billing Cost", + "operationType": "max", + "scale": "ratio", + "sourceField": "gcp.billing.total" + }, + "b92edf5e-58bc-4382-9cd5-19db2c332c93": { + "customLabel": true, + "dataType": "string", + "isBucketed": true, + "label": "Invoice Month", + "operationType": "terms", + "params": { + "orderBy": { + "type": "alphabetical" + }, + "orderDirection": "asc", + "size": 5 + }, + "scale": "ordinal", + "sourceField": "gcp.billing.invoice_month" + }, + "e25f49de-f161-4be8-a8fc-519188a7776c": { + "customLabel": true, + "dataType": "string", + "isBucketed": true, + "label": "Monthly Cost", + "operationType": "terms", + "params": { + "orderBy": { + "columnId": "af747bf6-66e9-4760-bbd8-3dae9c97159d", + "type": "column" + }, + "orderDirection": "desc", + "size": 10 + }, + "scale": "ordinal", + "sourceField": "gcp.billing.project_id" + } + } + } + } + } + }, + "filters": [], + "query": { + "language": "kuery", + "query": "" + }, + "visualization": { + "fittingFunction": "None", + "layers": [ + { + "accessors": [ + "af747bf6-66e9-4760-bbd8-3dae9c97159d" + ], + "layerId": "4ca843af-63d7-46b9-a719-51a81eebf1f7", + "layerType": "data", + "seriesType": "bar_stacked", + "splitAccessor": "b92edf5e-58bc-4382-9cd5-19db2c332c93", + "xAccessor": "e25f49de-f161-4be8-a8fc-519188a7776c" + } + ], + "legend": { + "isVisible": true, + "position": "right" + }, + "preferredSeriesType": "bar_stacked" + } + }, + "title": "Monthly Cost Per Project [Metrics GCP]", + "visualizationType": "lnsXY" + }, + "coreMigrationVersion": "7.15.0", + "id": "gcp-e6933020-e88d-11ea-bf8c-d13ebf358a78", + "migrationVersion": { + "lens": "7.15.0" + }, + "references": [ + { + "id": "metrics-*", + "name": "indexpattern-datasource-current-indexpattern", + "type": "index-pattern" + }, + { + "id": "metrics-*", + "name": "indexpattern-datasource-layer-4ca843af-63d7-46b9-a719-51a81eebf1f7", + "type": "index-pattern" + } + ], + "type": "lens" +} \ No newline at end of file diff --git a/packages/gcp/kibana/visualization/gcp-3f472ea0-5e47-11ea-a4f6-717338406083.json b/packages/gcp/kibana/visualization/gcp-3f472ea0-5e47-11ea-a4f6-717338406083.json new file mode 100644 index 00000000000..9534b5c85ad --- /dev/null +++ b/packages/gcp/kibana/visualization/gcp-3f472ea0-5e47-11ea-a4f6-717338406083.json @@ -0,0 +1,67 @@ +{ + "attributes": { + "description": "", + "kibanaSavedObjectMeta": { + "searchSourceJSON": {} + }, + "title": "Compute CPU Utilization [Metrics GCP]", + "uiStateJSON": {}, + "version": 1, + "visState": { + "aggs": [], + "params": { + "axis_formatter": "number", + "axis_position": "left", + "axis_scale": "normal", + "drop_last_bucket": 0, + "id": "61ca57f0-469d-11e7-af02-69e470af7417", + "index_pattern": "metrics-*", + "interval": "5m", + "isModelInvalid": false, + "series": [ + { + "axis_position": "right", + "chart_type": "line", + "color": "#68BC00", + "fill": "0", + "formatter": "percent", + "id": "61ca57f1-469d-11e7-af02-69e470af7417", + "label": "", + "line_width": "2", + "metrics": [ + { + "field": "gcp.compute.instance.cpu.usage.pct", + "id": "61ca57f2-469d-11e7-af02-69e470af7417", + "type": "avg" + } + ], + "point_size": "3", + "separate_axis": 0, + "split_color_mode": "rainbow", + "split_mode": "terms", + "stacked": "none", + "terms_field": "cloud.instance.name", + "terms_order_by": "61ca57f2-469d-11e7-af02-69e470af7417", + "type": "timeseries" + } + ], + "show_grid": 1, + "show_legend": 1, + "time_field": "", + "type": "timeseries", + "use_kibana_indexes": false + }, + "title": "Compute CPU Utilization [Metrics GCP]", + "type": "metrics" + } + }, + "coreMigrationVersion": "8.0.0", + "id": "gcp-3f472ea0-5e47-11ea-a4f6-717338406083", + "migrationVersion": { + "visualization": "7.14.0" + }, + "references": [], + "type": "visualization", + "updated_at": "2021-08-04T16:31:09.611Z", + "version": "WzM3NzcsMV0=" +} \ No newline at end of file diff --git a/packages/gcp/kibana/visualization/gcp-43f45ba0-5e4a-11ea-a4f6-717338406083.json b/packages/gcp/kibana/visualization/gcp-43f45ba0-5e4a-11ea-a4f6-717338406083.json new file mode 100644 index 00000000000..949d2470c8a --- /dev/null +++ b/packages/gcp/kibana/visualization/gcp-43f45ba0-5e4a-11ea-a4f6-717338406083.json @@ -0,0 +1,66 @@ +{ + "attributes": { + "description": "", + "kibanaSavedObjectMeta": { + "searchSourceJSON": {} + }, + "title": "Compute Network Received Bytes [Metrics GCP]", + "uiStateJSON": {}, + "version": 1, + "visState": { + "aggs": [], + "params": { + "axis_formatter": "number", + "axis_position": "left", + "axis_scale": "normal", + "drop_last_bucket": 0, + "id": "61ca57f0-469d-11e7-af02-69e470af7417", + "index_pattern": "metrics-*", + "interval": "5m", + "isModelInvalid": false, + "series": [ + { + "axis_position": "right", + "chart_type": "line", + "color": "#68BC00", + "fill": "0", + "formatter": "bytes", + "id": "61ca57f1-469d-11e7-af02-69e470af7417", + "label": "", + "line_width": "2", + "metrics": [ + { + "field": "gcp.compute.instance.network.ingress.bytes", + "id": "61ca57f2-469d-11e7-af02-69e470af7417", + "type": "avg" + } + ], + "point_size": "3", + "separate_axis": 0, + "split_color_mode": "rainbow", + "split_mode": "terms", + "stacked": "none", + "terms_field": "cloud.instance.name", + "type": "timeseries" + } + ], + "show_grid": 1, + "show_legend": 1, + "time_field": "", + "type": "timeseries", + "use_kibana_indexes": false + }, + "title": "Compute Network Received Bytes [Metrics GCP]", + "type": "metrics" + } + }, + "coreMigrationVersion": "8.0.0", + "id": "gcp-43f45ba0-5e4a-11ea-a4f6-717338406083", + "migrationVersion": { + "visualization": "7.14.0" + }, + "references": [], + "type": "visualization", + "updated_at": "2021-08-04T16:31:09.611Z", + "version": "WzM3ODEsMV0=" +} \ No newline at end of file diff --git a/packages/gcp/kibana/visualization/gcp-5d2f9160-e88e-11ea-bf8c-d13ebf358a78.json b/packages/gcp/kibana/visualization/gcp-5d2f9160-e88e-11ea-bf8c-d13ebf358a78.json new file mode 100644 index 00000000000..49d7087afb2 --- /dev/null +++ b/packages/gcp/kibana/visualization/gcp-5d2f9160-e88e-11ea-bf8c-d13ebf358a78.json @@ -0,0 +1,97 @@ +{ + "attributes": { + "description": "", + "kibanaSavedObjectMeta": { + "searchSourceJSON": { + "filter": [], + "query": { + "language": "kuery", + "query": "" + } + } + }, + "title": "Account ID Filter [Metrics GCP]", + "uiStateJSON": {}, + "version": 1, + "visState": { + "aggs": [], + "params": { + "controls": [ + { + "fieldName": "cloud.provider", + "id": "1598550838945", + "indexPatternRefName": "control_0_index_pattern", + "label": "Cloud Provider ", + "options": { + "dynamicOptions": true, + "multiselect": false, + "order": "desc", + "size": 5, + "type": "terms" + }, + "parent": "", + "type": "list" + }, + { + "fieldName": "cloud.account.id", + "id": "1598893530938", + "indexPatternRefName": "control_1_index_pattern", + "label": "Account ID", + "options": { + "dynamicOptions": true, + "multiselect": true, + "order": "desc", + "size": 5, + "type": "terms" + }, + "parent": "1598550838945", + "type": "list" + }, + { + "fieldName": "gcp.billing.invoice_month", + "id": "1598988595566", + "indexPatternRefName": "control_2_index_pattern", + "label": "Invoice Month", + "options": { + "dynamicOptions": true, + "multiselect": true, + "order": "desc", + "size": 5, + "type": "terms" + }, + "parent": "", + "type": "list" + } + ], + "pinFilters": false, + "updateFiltersOnChange": false, + "useTimeFilter": false + }, + "title": "Account ID Filter [Metrics GCP]", + "type": "input_control_vis" + } + }, + "coreMigrationVersion": "7.15.0", + "id": "gcp-5d2f9160-e88e-11ea-bf8c-d13ebf358a78", + "migrationVersion": { + "visualization": "7.14.0" + }, + "references": [ + { + "id": "metrics-*", + "name": "control_0_index_pattern", + "type": "index-pattern" + }, + { + "id": "metrics-*", + "name": "control_1_index_pattern", + "type": "index-pattern" + }, + { + "id": "metrics-*", + "name": "control_2_index_pattern", + "type": "index-pattern" + } + ], + "type": "visualization" +} \ No newline at end of file diff --git a/packages/gcp/kibana/visualization/gcp-6f795e70-5e49-11ea-a4f6-717338406083.json b/packages/gcp/kibana/visualization/gcp-6f795e70-5e49-11ea-a4f6-717338406083.json new file mode 100644 index 00000000000..ff34a41cd74 --- /dev/null +++ b/packages/gcp/kibana/visualization/gcp-6f795e70-5e49-11ea-a4f6-717338406083.json @@ -0,0 +1,66 @@ +{ + "attributes": { + "description": "", + "kibanaSavedObjectMeta": { + "searchSourceJSON": {} + }, + "title": "Compute Network Sent Bytes [Metrics GCP]", + "uiStateJSON": {}, + "version": 1, + "visState": { + "aggs": [], + "params": { + "axis_formatter": "number", + "axis_position": "left", + "axis_scale": "normal", + "drop_last_bucket": 0, + "id": "61ca57f0-469d-11e7-af02-69e470af7417", + "index_pattern": "metrics-*", + "interval": "5m", + "isModelInvalid": false, + "series": [ + { + "axis_position": "right", + "chart_type": "line", + "color": "#68BC00", + "fill": "0", + "formatter": "bytes", + "id": "61ca57f1-469d-11e7-af02-69e470af7417", + "label": "", + "line_width": "2", + "metrics": [ + { + "field": "gcp.compute.instance.network.egress.bytes", + "id": "61ca57f2-469d-11e7-af02-69e470af7417", + "type": "avg" + } + ], + "point_size": "3", + "separate_axis": 0, + "split_color_mode": "rainbow", + "split_mode": "terms", + "stacked": "none", + "terms_field": "cloud.instance.name", + "type": "timeseries" + } + ], + "show_grid": 1, + "show_legend": 1, + "time_field": "", + "type": "timeseries", + "use_kibana_indexes": false + }, + "title": "Compute Network Sent Bytes [Metrics GCP]", + "type": "metrics" + } + }, + "coreMigrationVersion": "8.0.0", + "id": "gcp-6f795e70-5e49-11ea-a4f6-717338406083", + "migrationVersion": { + "visualization": "7.14.0" + }, + "references": [], + "type": "visualization", + "updated_at": "2021-08-04T16:31:09.611Z", + "version": "WzM3ODAsMV0=" +} \ No newline at end of file diff --git a/packages/gcp/kibana/visualization/gcp-89513bc0-5e48-11ea-a4f6-717338406083.json b/packages/gcp/kibana/visualization/gcp-89513bc0-5e48-11ea-a4f6-717338406083.json new file mode 100644 index 00000000000..e57fde7b95d --- /dev/null +++ b/packages/gcp/kibana/visualization/gcp-89513bc0-5e48-11ea-a4f6-717338406083.json @@ -0,0 +1,66 @@ +{ + "attributes": { + "description": "", + "kibanaSavedObjectMeta": { + "searchSourceJSON": {} + }, + "title": "Compute Read I/O [Metrics GCP]", + "uiStateJSON": {}, + "version": 1, + "visState": { + "aggs": [], + "params": { + "axis_formatter": "number", + "axis_position": "left", + "axis_scale": "normal", + "drop_last_bucket": 0, + "id": "61ca57f0-469d-11e7-af02-69e470af7417", + "index_pattern": "metrics-*", + "interval": "5m", + "isModelInvalid": false, + "series": [ + { + "axis_position": "right", + "chart_type": "line", + "color": "#68BC00", + "fill": "0", + "formatter": "number", + "id": "61ca57f1-469d-11e7-af02-69e470af7417", + "label": "", + "line_width": "2", + "metrics": [ + { + "field": "gcp.compute.instance.disk.read_ops_count.value", + "id": "61ca57f2-469d-11e7-af02-69e470af7417", + "type": "avg" + } + ], + "point_size": "3", + "separate_axis": 0, + "split_color_mode": "rainbow", + "split_mode": "terms", + "stacked": "none", + "terms_field": "cloud.instance.name", + "type": "timeseries" + } + ], + "show_grid": 1, + "show_legend": 1, + "time_field": "", + "type": "timeseries", + "use_kibana_indexes": false + }, + "title": "Compute Read I/O [Metrics GCP]", + "type": "metrics" + } + }, + "coreMigrationVersion": "8.0.0", + "id": "gcp-89513bc0-5e48-11ea-a4f6-717338406083", + "migrationVersion": { + "visualization": "7.14.0" + }, + "references": [], + "type": "visualization", + "updated_at": "2021-08-04T16:31:09.611Z", + "version": "WzM3NzgsMV0=" +} \ No newline at end of file diff --git a/packages/gcp/kibana/visualization/gcp-95e1f050-5e48-11ea-a4f6-717338406083.json b/packages/gcp/kibana/visualization/gcp-95e1f050-5e48-11ea-a4f6-717338406083.json new file mode 100644 index 00000000000..3e1642e36bb --- /dev/null +++ b/packages/gcp/kibana/visualization/gcp-95e1f050-5e48-11ea-a4f6-717338406083.json @@ -0,0 +1,66 @@ +{ + "attributes": { + "description": "", + "kibanaSavedObjectMeta": { + "searchSourceJSON": {} + }, + "title": "Compute Write I/O [Metrics GCP]", + "uiStateJSON": {}, + "version": 1, + "visState": { + "aggs": [], + "params": { + "axis_formatter": "number", + "axis_position": "left", + "axis_scale": "normal", + "drop_last_bucket": 0, + "id": "61ca57f0-469d-11e7-af02-69e470af7417", + "index_pattern": "metrics-*", + "interval": "5m", + "isModelInvalid": false, + "series": [ + { + "axis_position": "right", + "chart_type": "line", + "color": "#68BC00", + "fill": "0", + "formatter": "number", + "id": "61ca57f1-469d-11e7-af02-69e470af7417", + "label": "", + "line_width": "2", + "metrics": [ + { + "field": "gcp.compute.instance.disk.write_ops_count.value", + "id": "61ca57f2-469d-11e7-af02-69e470af7417", + "type": "avg" + } + ], + "point_size": "3", + "separate_axis": 0, + "split_color_mode": "rainbow", + "split_mode": "terms", + "stacked": "none", + "terms_field": "cloud.instance.name", + "type": "timeseries" + } + ], + "show_grid": 1, + "show_legend": 1, + "time_field": "", + "type": "timeseries", + "use_kibana_indexes": false + }, + "title": "Compute Write I/O [Metrics GCP]", + "type": "metrics" + } + }, + "coreMigrationVersion": "8.0.0", + "id": "gcp-95e1f050-5e48-11ea-a4f6-717338406083", + "migrationVersion": { + "visualization": "7.14.0" + }, + "references": [], + "type": "visualization", + "updated_at": "2021-08-04T16:31:09.611Z", + "version": "WzM3NzksMV0=" +} \ No newline at end of file diff --git a/packages/gcp/kibana/visualization/gcp-9d919d00-5e4d-11ea-a4f6-717338406083.json b/packages/gcp/kibana/visualization/gcp-9d919d00-5e4d-11ea-a4f6-717338406083.json new file mode 100644 index 00000000000..6c0bfe94ed6 --- /dev/null +++ b/packages/gcp/kibana/visualization/gcp-9d919d00-5e4d-11ea-a4f6-717338406083.json @@ -0,0 +1,85 @@ +{ + "attributes": { + "description": "", + "kibanaSavedObjectMeta": { + "searchSourceJSON": {} + }, + "title": "Compute Firewall Dropped Bytes [Metrics GCP]", + "uiStateJSON": {}, + "version": 1, + "visState": { + "aggs": [], + "params": { + "axis_formatter": "number", + "axis_position": "left", + "axis_scale": "normal", + "background_color_rules": [ + { + "id": "3ece14c0-5e4c-11ea-9061-37f24ca5b01f" + } + ], + "bar_color_rules": [ + { + "id": "3b9c35c0-5e4c-11ea-9061-37f24ca5b01f" + } + ], + "drop_last_bucket": 0, + "gauge_color_rules": [ + { + "id": "3b27a200-5e4c-11ea-9061-37f24ca5b01f" + } + ], + "gauge_inner_width": 10, + "gauge_style": "half", + "gauge_width": 10, + "id": "61ca57f0-469d-11e7-af02-69e470af7417", + "index_pattern": "metrics-*", + "interval": "5m", + "isModelInvalid": false, + "series": [ + { + "axis_position": "right", + "chart_type": "line", + "color": "#68BC00", + "fill": "0", + "formatter": "bytes", + "id": "61ca57f1-469d-11e7-af02-69e470af7417", + "label": "", + "line_width": "2", + "metrics": [ + { + "field": "gcp.compute.firewall.dropped.bytes", + "id": "61ca57f2-469d-11e7-af02-69e470af7417", + "type": "avg" + } + ], + "point_size": "3", + "separate_axis": 0, + "split_color_mode": "rainbow", + "split_mode": "terms", + "stacked": "none", + "terms_field": "cloud.instance.name", + "terms_order_by": "61ca57f2-469d-11e7-af02-69e470af7417", + "type": "timeseries" + } + ], + "show_grid": 1, + "show_legend": 1, + "time_field": "", + "type": "timeseries", + "use_kibana_indexes": false + }, + "title": "Compute Firewall Dropped Bytes [Metrics GCP]", + "type": "metrics" + } + }, + "coreMigrationVersion": "8.0.0", + "id": "gcp-9d919d00-5e4d-11ea-a4f6-717338406083", + "migrationVersion": { + "visualization": "7.14.0" + }, + "references": [], + "type": "visualization", + "updated_at": "2021-08-04T16:31:09.611Z", + "version": "WzM3ODIsMV0=" +} \ No newline at end of file diff --git a/packages/gcp/kibana/visualization/gcp-da5bc460-63e1-11ea-b0ac-95d4ecb1fecd.json b/packages/gcp/kibana/visualization/gcp-da5bc460-63e1-11ea-b0ac-95d4ecb1fecd.json new file mode 100644 index 00000000000..1390eca96f0 --- /dev/null +++ b/packages/gcp/kibana/visualization/gcp-da5bc460-63e1-11ea-b0ac-95d4ecb1fecd.json @@ -0,0 +1,116 @@ +{ + "attributes": { + "description": "", + "kibanaSavedObjectMeta": { + "searchSourceJSON": {} + }, + "title": "Compute Instance Uptime Gauge [Metrics GCP]", + "uiStateJSON": {}, + "version": 1, + "visState": { + "aggs": [], + "params": { + "axis_formatter": "number", + "axis_position": "left", + "axis_scale": "normal", + "background_color_rules": [ + { + "id": "74a18260-63df-11ea-9543-55b68a4bcad3" + } + ], + "bar_color_rules": [ + { + "id": "77a54c80-63df-11ea-9543-55b68a4bcad3" + } + ], + "drop_last_bucket": 0, + "filter": { + "language": "kuery", + "query": "" + }, + "gauge_color_rules": [ + { + "id": "777371a0-63e0-11ea-9543-55b68a4bcad3", + "value": 0 + } + ], + "gauge_inner_width": 10, + "gauge_max": "", + "gauge_style": "circle", + "gauge_width": 10, + "hide_last_value_indicator": true, + "id": "61ca57f0-469d-11e7-af02-69e470af7417", + "index_pattern": "metrics-*", + "interval": "\u003e=5m", + "isModelInvalid": false, + "series": [ + { + "axis_position": "right", + "chart_type": "bar", + "color": "#68BC00", + "fill": 0.5, + "filter": { + "language": "kuery", + "query": "" + }, + "formatter": "percent", + "id": "61ca57f1-469d-11e7-af02-69e470af7417", + "label": "Average Uptime", + "line_width": 1, + "metrics": [ + { + "denominator": "60", + "field": "gcp.compute.instance.uptime.sec", + "id": "61ca57f2-469d-11e7-af02-69e470af7417", + "numerator": "gcp.compute.instance.uptime.sec", + "type": "avg", + "values": [ + "60" + ] + }, + { + "id": "81dc6000-63e7-11ea-994d-3b2599babc53", + "script": "params.uptime / 60\n", + "type": "math", + "variables": [ + { + "field": "61ca57f2-469d-11e7-af02-69e470af7417", + "id": "85f3bd00-63e7-11ea-994d-3b2599babc53", + "name": "uptime" + } + ] + } + ], + "override_index_pattern": 0, + "point_size": 1, + "separate_axis": 0, + "split_color_mode": "gradient", + "split_mode": "everything", + "stacked": "none", + "terms_field": "cloud.instance.name", + "terms_order_by": "61ca57f2-469d-11e7-af02-69e470af7417", + "type": "timeseries" + } + ], + "show_grid": 1, + "show_legend": 1, + "time_field": "@timestamp", + "time_range_mode": "entire_time_range", + "tooltip_mode": "show_all", + "type": "gauge", + "use_kibana_indexes": false + }, + "title": "Compute Instance Uptime Gauge [Metrics GCP]", + "type": "metrics" + } + }, + "coreMigrationVersion": "8.0.0", + "id": "gcp-da5bc460-63e1-11ea-b0ac-95d4ecb1fecd", + "migrationVersion": { + "visualization": "7.14.0" + }, + "references": [], + "type": "visualization", + "updated_at": "2021-08-04T16:31:09.611Z", + "version": "WzM3NzYsMV0=" +} \ No newline at end of file diff --git a/packages/gcp/kibana/visualization/gcp-ef1508c0-5e4c-11ea-a4f6-717338406083.json b/packages/gcp/kibana/visualization/gcp-ef1508c0-5e4c-11ea-a4f6-717338406083.json new file mode 100644 index 00000000000..61e2e9586c1 --- /dev/null +++ b/packages/gcp/kibana/visualization/gcp-ef1508c0-5e4c-11ea-a4f6-717338406083.json @@ -0,0 +1,87 @@ +{ + "attributes": { + "description": "", + "kibanaSavedObjectMeta": { + "searchSourceJSON": {} + }, + "title": "Compute Firewall Dropped Packets [Metrics GCP]", + "uiStateJSON": {}, + "version": 1, + "visState": { + "aggs": [], + "params": { + "axis_formatter": "number", + "axis_position": "left", + "axis_scale": "normal", + "background_color_rules": [ + { + "id": "3ece14c0-5e4c-11ea-9061-37f24ca5b01f" + } + ], + "bar_color_rules": [ + { + "id": "3b9c35c0-5e4c-11ea-9061-37f24ca5b01f" + } + ], + "drop_last_bucket": 0, + "gauge_color_rules": [ + { + "id": "3b27a200-5e4c-11ea-9061-37f24ca5b01f" + } + ], + "gauge_inner_width": 10, + "gauge_style": "half", + "gauge_width": 10, + "hide_last_value_indicator": true, + "id": "61ca57f0-469d-11e7-af02-69e470af7417", + "index_pattern": "metrics-*", + "interval": "5m", + "isModelInvalid": false, + "series": [ + { + "axis_position": "right", + "chart_type": "line", + "color": "#68BC00", + "fill": "0", + "formatter": "number", + "id": "61ca57f1-469d-11e7-af02-69e470af7417", + "label": "", + "line_width": "3", + "metrics": [ + { + "field": "gcp.compute.firewall.dropped_packets_count.value", + "id": "61ca57f2-469d-11e7-af02-69e470af7417", + "type": "avg" + } + ], + "point_size": "2", + "separate_axis": 0, + "split_color_mode": "rainbow", + "split_mode": "terms", + "stacked": "none", + "terms_field": "cloud.instance.name", + "terms_order_by": "61ca57f2-469d-11e7-af02-69e470af7417", + "type": "timeseries" + } + ], + "show_grid": 1, + "show_legend": 1, + "time_field": "", + "time_range_mode": "entire_time_range", + "type": "top_n", + "use_kibana_indexes": false + }, + "title": "Compute Firewall Dropped Packets [Metrics GCP]", + "type": "metrics" + } + }, + "coreMigrationVersion": "8.0.0", + "id": "gcp-ef1508c0-5e4c-11ea-a4f6-717338406083", + "migrationVersion": { + "visualization": "7.14.0" + }, + "references": [], + "type": "visualization", + "updated_at": "2021-08-04T16:31:09.611Z", + "version": "WzM3ODMsMV0=" +} \ No newline at end of file diff --git a/packages/gcp/manifest.yml b/packages/gcp/manifest.yml index d5e012c4352..f3c46db1828 100644 --- a/packages/gcp/manifest.yml +++ b/packages/gcp/manifest.yml @@ -1,6 +1,6 @@ name: gcp title: Google Cloud Platform -version: "1.10.0" +version: 2.0.0 release: ga description: Collect logs from Google Cloud Platform with Elastic Agent. type: integration @@ -14,51 +14,124 @@ license: basic categories: - google_cloud - cloud - - network - - security conditions: - kibana.version: ^7.17.0 || ^8.0.0 + kibana.version: ^8.3.0 screenshots: - src: /img/filebeat-gcp-audit.png title: filebeat gcp audit size: 1702x996 type: image/png + - src: /img/gcp-billing.png + title: GCP Billing Metrics Dashboard + size: 2000x1020 + type: image/png + - src: /img/gcp-compute.png + title: GCP Compute Metrics Dashboard + size: 2000x2021 + type: image/png +vars: + - name: project_id + type: text + title: Project Id + multi: false + required: true + show_user: true + default: SET_PROJECT_NAME + - name: credentials_file + type: text + title: Credentials File + multi: false + required: false + show_user: true + - name: credentials_json + type: text + title: Credentials Json + multi: false + required: false + show_user: true policy_templates: - - name: gcp - title: Google Cloud Platform (GCP) logs - description: Collect logs from Google Cloud Platform (GCP) instances + - name: audit + title: Google Cloud Platform (GCP) Audit logs + description: Collect audit logs from Google Cloud Platform (GCP) with Elastic Agent + categories: + - security + data_streams: + - audit inputs: - type: gcp-pubsub - vars: - - name: alternative_host - type: text - title: Alternative host - multi: false - required: false - show_user: false - - name: project_id - type: text - title: Project Id - description: Your Google Cloud project ID where the resources exist. - multi: false - required: true - show_user: true - default: SET_PROJECT_NAME - - name: credentials_file - type: text - title: Credentials File - description: The path to the JSON file with the private key. Make sure that the Elastic Agent has at least read-only privileges to this file. - multi: false - required: false - show_user: true - - name: credentials_json - type: text - title: Credentials JSON - description: The content of the JSON file you downloaded from Google Cloud Platform. - multi: false - required: false - show_user: true - title: "Collect Google Cloud Platform (GCP) audit, firewall and vpcflow logs (input: gcp-pubsub)" - description: "Collecting audit, firewall and vpcflow logs from Google Cloud Platform (GCP) instances (input: gcp-pubsub)" + title: "Collect Google Cloud Platform (GCP) audit logs (input: gcp-pubsub)" + description: "Collecting audit logs from Google Cloud Platform (GCP) instances (input: gcp-pubsub)" + input_group: logs + screenshots: + - src: /img/filebeat-gcp-audit.png + title: filebeat gcp audit + size: 1702x996 + type: image/png + - name: firewall + title: Google Cloud Platform (GCP) Firewall logs + description: Collect firewall logs from Google Cloud Platform (GCP) with Elastic Agent + categories: + - network + - security + data_streams: + - firewall + inputs: + - type: gcp-pubsub + title: "Collect Google Cloud Platform (GCP) firewall logs (input: gcp-pubsub)" + description: "Collecting firewall logs from Google Cloud Platform (GCP) instances (input: gcp-pubsub)" + input_group: logs + - name: vpcflow + title: Google Cloud Platform (GCP) VPC Flow logs + description: Collect vpcflow logs from Google Cloud Platform (GCP) with Elastic Agent + categories: + - network + - security + data_streams: + - vpcflow + inputs: + - type: gcp-pubsub + title: "Collect Google Cloud Platform (GCP) VPC Flow logs (input: gcp-pubsub)" + description: "Collecting vpcflow logs from Google Cloud Platform (GCP) instances (input: gcp-pubsub)" + input_group: logs + - name: billing + title: Google Cloud Platform (GCP) Billine metrics + description: Collect billing metrics from Google Cloud Platform (GCP) with Elastic Agent + data_streams: + - billing + inputs: + - type: gcp/metrics + title: Collect GCP Billing Metrics + description: Collect GCP Billing Metrics + input_group: metrics + screenshots: + - src: /img/gcp-billing.png + title: GCP Billing Metrics Dashboard + size: 2000x1020 + type: image/png + - name: compute + title: Google Cloud Platform (GCP) Compute metrics + description: Collect compute metrics from Google Cloud Platform (GCP) with Elastic Agent + data_streams: + - compute + inputs: + - type: gcp/metrics + title: Collect GCP Compute Metrics + description: Collect GCP Compute Metrics + input_group: metrics + screenshots: + - src: /img/gcp-compute.png + title: GCP Compute Metrics Dashboard + size: 2000x2021 + type: image/png + - name: firestore + title: Google Cloud Platform (GCP) Firestore metrics + description: Collect firestore metrics from Google Cloud Platform (GCP) with Elastic Agent + data_streams: + - firestore + inputs: + - type: gcp/metrics + title: Collect GCP Firestore Metrics + description: Collect GCP Firestore Metrics + input_group: metrics owner: github: elastic/security-external-integrations