Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[GCP] Add firestore data_stream #2704

Merged
merged 28 commits into from
Mar 30, 2022
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
28 commits
Select commit Hold shift + click to select a range
7c53646
gcp: stub logs and metrics doc files
endorama Nov 8, 2021
418224b
[GCP] Add GCP Billing Metricset (#2141)
legoguy1000 Nov 30, 2021
6a2821f
[GCP] Add GCP Compute Metricset (#2301)
legoguy1000 Dec 31, 2021
e3cfa71
[GCP] Add GCP Billing Metricset (#2141)
legoguy1000 Nov 30, 2021
3e4354e
[GCP] Add GCP Compute Metricset (#2301)
legoguy1000 Dec 31, 2021
ad4e9b9
Add firestore data_stream
gpop63 Feb 16, 2022
417bfbe
Fix formatting issue
gpop63 Feb 16, 2022
ceea735
Add gcp tests env vars in Jenkinsfile
gpop63 Mar 16, 2022
a0a8e1e
Add missing env fields
gpop63 Mar 16, 2022
1b1db81
Update terraform deployer
gpop63 Mar 16, 2022
62cb4d6
Update system test
gpop63 Mar 16, 2022
a07b316
Fix credentials json field
gpop63 Mar 16, 2022
d1d3ac7
Format & add period
gpop63 Mar 16, 2022
c7f878d
Update docs
gpop63 Mar 17, 2022
4186f6b
Update default period for firestoer
gpop63 Mar 17, 2022
bb4f48f
Update terraform deployer
gpop63 Mar 17, 2022
f04b736
Remove period from system test
gpop63 Mar 17, 2022
be8a0ae
Update manifest version
gpop63 Mar 17, 2022
55718c9
Update manifest file
gpop63 Mar 17, 2022
2be9cb5
Add default project id
gpop63 Mar 17, 2022
9ddd3f6
Fix owner
gpop63 Mar 21, 2022
957e4f3
Update audit docs
gpop63 Mar 21, 2022
21e114f
Fix terraform deployer
gpop63 Mar 22, 2022
de949e9
Remove redundant alternative host
gpop63 Mar 23, 2022
e5a0004
Fix project variable
gpop63 Mar 23, 2022
b8bc2c0
Remove default project string
gpop63 Mar 23, 2022
9f6be59
Remove firestore datastream tests
gpop63 Mar 30, 2022
9ed8658
Revert "Add gcp tests env vars in Jenkinsfile"
gpop63 Mar 30, 2022
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 12 additions & 0 deletions packages/gcp/_dev/build/docs/audit.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
# Audit

Google Cloud services write audit logs that record administrative activities and accesses within your Google Cloud resources.
More information on the type of data included in the audit logs can be found in the [documentation](https://cloud.google.com/logging/docs/audit)

## Logs

This is the `audit` dataset.

{{event "audit"}}

{{fields "audit"}}
9 changes: 9 additions & 0 deletions packages/gcp/_dev/build/docs/billing.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
# Billing

## Metrics

This is the `billing` dataset.

{{event "billing"}}

{{fields "billing"}}
9 changes: 9 additions & 0 deletions packages/gcp/_dev/build/docs/firestore.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
# Firestore

## Metrics

This is the `firestore` dataset.

{{event "firestore"}}

{{fields "firestore"}}
12 changes: 12 additions & 0 deletions packages/gcp/_dev/build/docs/firewall.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
# Firewall

Firewall Rules Logging lets you audit, verify, and analyze the effects of your firewall rules.
More information on the type of data included in the firewall logs can be found in the [documentation](https://cloud.google.com/vpc/docs/firewall-rules-logging)

## Logs

This is the `firewall` dataset.

{{event "firewall"}}

{{fields "firewall"}}
13 changes: 13 additions & 0 deletions packages/gcp/_dev/build/docs/vpcflow.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
# VPC Flow

VPC Flow Logs records a sample of network flows sent from and received by VM instances, including instances used as GKE nodes.
These logs can be used for network monitoring, forensics, real-time security analysis, and expense optimization.
More information on the type of data included in the firewall logs can be found in the [documentation](https://cloud.google.com/vpc/docs/using-flow-logs)

## Logs

This is the `vpcflow` dataset.

{{event "vpcflow"}}

{{fields "vpcflow"}}
5 changes: 5 additions & 0 deletions packages/gcp/changelog.yml
Original file line number Diff line number Diff line change
@@ -1,4 +1,9 @@
# newer versions go on top
- version: "1.4.3"
changes:
- description: Add GCP Firestore metricset
type: enhancement
link: https://github.com/elastic/integrations/pull/2704
- version: "1.4.2"
changes:
- description: Remove emtpy values, names with only dots, and invalid client IPs.
Expand Down
16 changes: 16 additions & 0 deletions packages/gcp/data_stream/firestore/agent/stream/stream.yml.hbs
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
metricsets: ["firestore"]
period: {{period}}
project_id: {{project_id}}
{{#if credentials_file}}
credentials_file_path: {{credentials_file}}
{{/if}}
{{#if credentials_json}}
credentials_json: '{{credentials_json}}'
{{/if}}
{{#if region}}
region: {{region}}
{{/if}}
{{#if zone}}
zone: {{zone}}
{{/if}}
exclude_labels: {{exclude_labels}}
198 changes: 198 additions & 0 deletions packages/gcp/data_stream/firestore/fields/agent.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,198 @@
- name: cloud
title: Cloud
group: 2
description: Fields related to the cloud or infrastructure the events are coming from.
footnote: 'Examples: If Metricbeat is running on an GCP Compute VM and fetches data from its host, the cloud info contains the data about this machine. If Metricbeat runs on a remote machine outside the cloud and fetches data from a service running in the cloud, the field contains cloud data from the machine the service is running on.'
type: group
fields:
- name: account.id
level: extended
type: keyword
ignore_above: 1024
description: 'The cloud account or organization id used to identify different entities in a multi-tenant environment.

Examples: AWS account id, Google Cloud ORG Id, or other unique identifier.'
example: 666777888999
- name: availability_zone
level: extended
type: keyword
ignore_above: 1024
description: Availability zone in which this host is running.
example: us-east-1c
- name: instance.id
level: extended
type: keyword
ignore_above: 1024
description: Instance ID of the host machine.
example: i-1234567890abcdef0
- name: instance.name
level: extended
type: keyword
ignore_above: 1024
description: Instance name of the host machine.
- name: machine.type
level: extended
type: keyword
ignore_above: 1024
description: Machine type of the host machine.
example: t2.medium
- name: provider
level: extended
type: keyword
ignore_above: 1024
description: Name of the cloud provider. Example values are aws, azure, gcp, or digitalocean.
example: aws
- name: region
level: extended
type: keyword
ignore_above: 1024
description: Region in which this host is running.
example: us-east-1
- name: project.id
type: keyword
description: Name of the project in Google Cloud.
- name: image.id
type: keyword
description: Image ID for the cloud instance.
- name: container
title: Container
group: 2
description: 'Container fields are used for meta information about the specific container that is the source of information.

These fields help correlate data based containers from any runtime.'
type: group
fields:
- name: id
level: core
type: keyword
ignore_above: 1024
description: Unique container id.
- name: image.name
level: extended
type: keyword
ignore_above: 1024
description: Name of the image the container was built on.
- name: labels
level: extended
type: object
object_type: keyword
description: Image labels.
- name: name
level: extended
type: keyword
ignore_above: 1024
description: Container name.
- name: host
title: Host
group: 2
description: 'A host is defined as a general computing instance.

ECS host.* fields should be populated with details about the host on which the event happened, or from which the measurement was taken. Host types include hardware, virtual machines, Docker containers, and Kubernetes nodes.'
type: group
fields:
- name: architecture
level: core
type: keyword
ignore_above: 1024
description: Operating system architecture.
example: x86_64
- name: domain
level: extended
type: keyword
ignore_above: 1024
description: 'Name of the domain of which the host is a member.

For example, on Windows this could be the host''s Active Directory domain or NetBIOS domain name. For Linux this could be the domain of the host''s LDAP provider.'
example: CONTOSO
default_field: false
- name: hostname
level: core
type: keyword
ignore_above: 1024
description: 'Hostname of the host.

It normally contains what the `hostname` command returns on the host machine.'
- name: id
level: core
type: keyword
ignore_above: 1024
description: 'Unique host id.

As hostname is not always unique, use values that are meaningful in your environment.

Example: The current usage of `beat.name`.'
- name: ip
level: core
type: ip
description: Host ip addresses.
- name: mac
level: core
type: keyword
ignore_above: 1024
description: Host mac addresses.
- name: name
level: core
type: keyword
ignore_above: 1024
description: 'Name of the host.

It can contain what `hostname` returns on Unix systems, the fully qualified domain name, or a name specified by the user. The sender decides which value to use.'
- name: os.family
level: extended
type: keyword
ignore_above: 1024
description: OS family (such as redhat, debian, freebsd, windows).
example: debian
- name: os.kernel
level: extended
type: keyword
ignore_above: 1024
description: Operating system kernel version as a raw string.
example: 4.4.0-112-generic
- name: os.name
level: extended
type: keyword
ignore_above: 1024
multi_fields:
- name: text
type: text
norms: false
default_field: false
description: Operating system name, without the version.
example: Mac OS X
- name: os.platform
level: extended
type: keyword
ignore_above: 1024
description: Operating system platform (such centos, ubuntu, windows).
example: darwin
- name: os.version
level: extended
type: keyword
ignore_above: 1024
description: Operating system version as a raw string.
example: 10.14.1
- name: type
level: core
type: keyword
ignore_above: 1024
description: 'Type of host.

For Cloud providers this can be the machine type like `t2.medium`. If vm, this could be the container, for example, or other information meaningful in your environment.'
- name: containerized
type: boolean
description: >
If the host is a container.

- name: os.build
type: keyword
example: "18D109"
description: >
OS build information.

- name: os.codename
type: keyword
example: "stretch"
description: >
OS codename, if any.

20 changes: 20 additions & 0 deletions packages/gcp/data_stream/firestore/fields/base-fields.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
- name: data_stream.type
type: constant_keyword
description: Data stream type.
- name: data_stream.dataset
type: constant_keyword
description: Data stream dataset.
- name: data_stream.namespace
type: constant_keyword
description: Data stream namespace.
- name: '@timestamp'
type: date
description: Event timestamp.
- name: event.module
type: constant_keyword
description: Event module
value: gcp
- name: event.dataset
type: constant_keyword
description: Event dataset
value: gcp.firestore
24 changes: 24 additions & 0 deletions packages/gcp/data_stream/firestore/fields/ecs.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
- external: ecs
name: cloud
- external: ecs
name: cloud.account.id
- external: ecs
name: cloud.account.name
- external: ecs
name: cloud.availability_zone
- external: ecs
name: cloud.instance.id
- external: ecs
name: cloud.machine.type
- external: ecs
name: cloud.provider
- external: ecs
name: cloud.region
- external: ecs
name: ecs.version
- external: ecs
name: error
- external: ecs
name: error.message
- external: ecs
name: service.type
13 changes: 13 additions & 0 deletions packages/gcp/data_stream/firestore/fields/fields.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
- name: gcp.firestore
description: Google Cloud Firestore metrics
type: group
fields:
- name: document.delete.count
type: long
description: The number of successful document deletes.
- name: document.read.count
type: long
description: The number of successful document reads from queries or lookups.
- name: document.write.count
type: long
description: The number of successful document writes.
31 changes: 31 additions & 0 deletions packages/gcp/data_stream/firestore/fields/package-fields.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
- name: gcp
description: >-
GCP module
fields:
- name: labels
type: object
description: >-
GCP monitoring metrics labels
fields:
- name: user.*
type: object
object_type: keyword
- name: metadata.*
type: object
object_type: keyword
- name: metrics.*
type: object
object_type: keyword
- name: system.*
type: object
object_type: keyword
- name: resource.*
type: object
object_type: keyword
- name: "metrics.*.*.*.*"
type: object
object_type: double
object_type_mapping_type: "*"
description: >
Metrics that returned from Google Cloud API query.

Loading