Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

SPLAT-1844: added step to report iam used by AWS job #58651

Draft
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

mtulio
Copy link
Contributor

@mtulio mtulio commented Nov 8, 2024

WIP MVP for https://issues.redhat.com/browse/SPLAT-1844

ci-operator/step-registry/gather/cloud-iam-access/aws/gather-cloud-iam-access-aws-commands.sh

This PR introduces a step to capture the audit logs from AWS, parse the events related to the CI job, and provide a overview comparing with the expected (requested by components).

@openshift-ci-robot
Copy link
Contributor

openshift-ci-robot commented Nov 8, 2024

@mtulio: This pull request references SPLAT-1844 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.18.0" version, but no target version was set.

In response to this:

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-ci-robot openshift-ci-robot added the jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. label Nov 8, 2024
@openshift-ci openshift-ci bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Nov 8, 2024
Copy link
Contributor

openshift-ci bot commented Nov 8, 2024

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

@openshift-ci openshift-ci bot added the do-not-merge/invalid-owners-file Indicates that a PR should not merge because it has an invalid OWNERS file in it. label Nov 8, 2024
@openshift-ci-robot
Copy link
Contributor

openshift-ci-robot commented Nov 8, 2024

@mtulio: This pull request references SPLAT-1844 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.18.0" version, but no target version was set.

In response to this:

WIP MVP for https://issues.redhat.com/browse/SPLAT-1844

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@mtulio
Copy link
Contributor Author

mtulio commented Nov 8, 2024

/pj-rehearse periodic-ci-openshift-release-master-nightly-4.18-e2e-aws-ovn-perms-discovery

@openshift-ci-robot
Copy link
Contributor

@mtulio: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

@mtulio mtulio force-pushed the mvp-cloud-iam-access branch from f95f0b7 to 5709c35 Compare November 8, 2024 17:16
@mtulio
Copy link
Contributor Author

mtulio commented Nov 8, 2024

fixed the logic to prevent reaching installer flag that is not yet delivered.

/pj-rehearse periodic-ci-openshift-release-master-nightly-4.18-e2e-aws-ovn-perms-discovery

@openshift-ci-robot
Copy link
Contributor

@mtulio: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

@mtulio mtulio force-pushed the mvp-cloud-iam-access branch 2 times, most recently from b46630b to 6a4195f Compare November 8, 2024 17:53
@openshift-ci openshift-ci bot removed the do-not-merge/invalid-owners-file Indicates that a PR should not merge because it has an invalid OWNERS file in it. label Nov 8, 2024
@mtulio
Copy link
Contributor Author

mtulio commented Nov 8, 2024

fixed the IAM create user step to prevent skipping when the policy file is not set - this workflow expect to skip the policy creation when using the managed one

/pj-rehearse periodic-ci-openshift-release-master-nightly-4.18-e2e-aws-ovn-perms-discovery

@openshift-ci-robot
Copy link
Contributor

@mtulio: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

@mtulio mtulio force-pushed the mvp-cloud-iam-access branch from 6a4195f to ab083f8 Compare November 8, 2024 18:02
@mtulio
Copy link
Contributor Author

mtulio commented Nov 8, 2024

/pj-rehearse periodic-ci-openshift-release-master-nightly-4.18-e2e-aws-ovn-perms-discovery

@openshift-ci-robot
Copy link
Contributor

@mtulio: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

@mtulio mtulio force-pushed the mvp-cloud-iam-access branch 2 times, most recently from 693c9a4 to 5cbb75a Compare November 8, 2024 21:53
@mtulio
Copy link
Contributor Author

mtulio commented Nov 8, 2024

/pj-rehearse periodic-ci-openshift-release-master-nightly-4.18-e2e-aws-ovn-perms-discovery

@openshift-ci-robot
Copy link
Contributor

@mtulio: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

@mtulio mtulio force-pushed the mvp-cloud-iam-access branch from 5cbb75a to 8c21511 Compare November 9, 2024 01:01
@mtulio
Copy link
Contributor Author

mtulio commented Nov 9, 2024

/pj-rehearse periodic-ci-openshift-release-master-nightly-4.18-e2e-aws-ovn-perms-discovery

For some reason the global vars CLUSTER_NAME isn't set in gather step. Setting it in initial steps to be reused by gather:

/pj-rehearse periodic-ci-openshift-release-master-nightly-4.18-e2e-aws-ovn-perms-discovery

@openshift-ci-robot
Copy link
Contributor

@mtulio: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

@mtulio
Copy link
Contributor Author

mtulio commented Nov 9, 2024

/pj-rehearse periodic-ci-openshift-release-master-nightly-4.18-e2e-aws-ovn-perms-discovery

@openshift-ci-robot
Copy link
Contributor

@mtulio: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

@mtulio mtulio force-pushed the mvp-cloud-iam-access branch from 8c21511 to f2789fa Compare November 11, 2024 14:40
@mtulio
Copy link
Contributor Author

mtulio commented Nov 11, 2024

Trying to ensure CLUSTER_NAME from shared dir:

/pj-rehearse periodic-ci-openshift-release-master-nightly-4.18-e2e-aws-ovn-perms-discovery

@mtulio
Copy link
Contributor Author

mtulio commented Jan 6, 2025

Added the exception of identifying cases in the policy name and report it in the diff section, allowing reviewers to identifying typos in the CredentialRequests.

/pj-rehearse periodic-ci-openshift-release-master-nightly-4.18-e2e-aws-ovn-audit-perms

@openshift-ci-robot
Copy link
Contributor

@mtulio: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

@mtulio
Copy link
Contributor Author

mtulio commented Jan 7, 2025

/test all

@mtulio mtulio changed the title DNM/WIP: SPLAT-1844: added step to report iam used by AWS job SPLAT-1844: added step to report iam used by AWS job Jan 7, 2025
@openshift-ci-robot
Copy link
Contributor

openshift-ci-robot commented Jan 7, 2025

@mtulio: This pull request references SPLAT-1844 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.19.0" version, but no target version was set.

In response to this:

WIP MVP for https://issues.redhat.com/browse/SPLAT-1844

ci-operator/step-registry/gather/cloud-iam-access/aws/gather-cloud-iam-access-aws-commands.sh

This PR introduces a step to capture the audit logs from AWS, parse the events related to the CI job, and provide a overview comparing with the expected (requested by components).

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@mtulio mtulio force-pushed the mvp-cloud-iam-access branch from 81c2c41 to 1347631 Compare January 8, 2025 15:26
@openshift-merge-robot openshift-merge-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jan 8, 2025
@mtulio mtulio force-pushed the mvp-cloud-iam-access branch from 1347631 to c927a3b Compare January 8, 2025 15:30
@openshift-merge-robot openshift-merge-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jan 8, 2025
@mtulio
Copy link
Contributor Author

mtulio commented Jan 8, 2025

/test all

@mtulio mtulio force-pushed the mvp-cloud-iam-access branch from c927a3b to 244ae22 Compare January 8, 2025 17:04
@mtulio
Copy link
Contributor Author

mtulio commented Jan 8, 2025

/test release-controller-config

@mtulio
Copy link
Contributor Author

mtulio commented Jan 8, 2025

/test all

Comment on lines +99 to +103
# TODO(mtulio): define where to save that cross-component tool to parse IAM events.
# This script must not be saved in component repo as it is intented to be used by cross
# repo on CI.
log_msg "Downloading cci (cloud credential insights) utility"
wget -qO $CCI https://raw.githubusercontent.com/openshift-splat-team/cloud-credentials-insights/refs/heads/devel-cci-aws/cci.py
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

merge this script to main branch before moving this job/step.

@mtulio mtulio force-pushed the mvp-cloud-iam-access branch from 244ae22 to bb52550 Compare January 9, 2025 01:16
@mtulio
Copy link
Contributor Author

mtulio commented Jan 9, 2025

/test release-controller-config

@mtulio
Copy link
Contributor Author

mtulio commented Jan 9, 2025

/test all

This PR introduces a step `cloud-iam-access`
to capture the audit logs from AWS, parse the
events related to the CI periodic
job `e2e-aws-ovn-audit-perms` - also introduced here,
providing a overview comparing with the
expected (requested by components).
@mtulio mtulio force-pushed the mvp-cloud-iam-access branch from bb52550 to d34d982 Compare January 9, 2025 14:45
@openshift-ci-robot
Copy link
Contributor

[REHEARSALNOTIFIER]
@mtulio: the pj-rehearse plugin accommodates running rehearsal tests for the changes in this PR. Expand 'Interacting with pj-rehearse' for usage details. The following rehearsable tests have been affected by this change:

Test name Repo Type Reason
pull-ci-openshift-assisted-service-master-okd-scos-e2e-aws-ovn openshift/assisted-service presubmit Registry content changed
pull-ci-openshift-assisted-service-release-4.18-okd-scos-e2e-aws-ovn openshift/assisted-service presubmit Registry content changed
pull-ci-kubevirt-vm-console-proxy-main-e2e-functests kubevirt/vm-console-proxy presubmit Registry content changed
pull-ci-kubevirt-vm-console-proxy-release-v0.1-e2e-functests kubevirt/vm-console-proxy presubmit Registry content changed
pull-ci-kubevirt-vm-console-proxy-release-v0.2-e2e-functests kubevirt/vm-console-proxy presubmit Registry content changed
pull-ci-kubevirt-vm-console-proxy-release-v0.5-e2e-functests kubevirt/vm-console-proxy presubmit Registry content changed
pull-ci-kubevirt-vm-console-proxy-stable-e2e-functests kubevirt/vm-console-proxy presubmit Registry content changed
pull-ci-kubevirt-vm-console-proxy-release-v0.6-e2e-functests kubevirt/vm-console-proxy presubmit Registry content changed
pull-ci-kubevirt-vm-console-proxy-release-v0.4-e2e-functests kubevirt/vm-console-proxy presubmit Registry content changed
pull-ci-kubevirt-vm-console-proxy-release-v0.3-e2e-functests kubevirt/vm-console-proxy presubmit Registry content changed
pull-ci-openshift-oadp-operator-master-4.16-e2e-test-aws openshift/oadp-operator presubmit Registry content changed
pull-ci-openshift-oadp-operator-master-4.16-e2e-test-kubevirt-aws openshift/oadp-operator presubmit Registry content changed
pull-ci-openshift-oadp-operator-master-4.18-dev-preview-e2e-test-aws openshift/oadp-operator presubmit Registry content changed
pull-ci-openshift-oadp-operator-master-4.17-e2e-test-aws openshift/oadp-operator presubmit Registry content changed
pull-ci-openshift-oadp-operator-master-4.17-e2e-test-kubevirt-aws openshift/oadp-operator presubmit Registry content changed
pull-ci-openshift-oadp-operator-master-4.14-e2e-test-aws openshift/oadp-operator presubmit Registry content changed
pull-ci-openshift-oadp-operator-master-4.14-e2e-test-kubevirt-aws openshift/oadp-operator presubmit Registry content changed
pull-ci-openshift-oadp-operator-master-4.15-e2e-test-aws openshift/oadp-operator presubmit Registry content changed
pull-ci-openshift-oadp-operator-master-4.15-e2e-test-kubevirt-aws openshift/oadp-operator presubmit Registry content changed
pull-ci-openshift-oadp-operator-oadp-1.4-4.15-e2e-test-aws openshift/oadp-operator presubmit Registry content changed
pull-ci-openshift-oadp-operator-oadp-1.4-4.15-e2e-test-kubevirt-aws openshift/oadp-operator presubmit Registry content changed
pull-ci-openshift-oadp-operator-oadp-1.4-4.16-e2e-test-aws openshift/oadp-operator presubmit Registry content changed
pull-ci-openshift-oadp-operator-oadp-1.4-4.16-e2e-test-kubevirt-aws openshift/oadp-operator presubmit Registry content changed
pull-ci-openshift-oadp-operator-oadp-1.4-4.14-e2e-test-aws openshift/oadp-operator presubmit Registry content changed
pull-ci-openshift-oadp-operator-oadp-1.4-4.14-e2e-test-kubevirt-aws openshift/oadp-operator presubmit Registry content changed

A total of 21963 jobs have been affected by this change. The above listing is non-exhaustive and limited to 25 jobs.

A full list of affected jobs can be found here

Interacting with pj-rehearse

Comment: /pj-rehearse to run up to 5 rehearsals
Comment: /pj-rehearse skip to opt-out of rehearsals
Comment: /pj-rehearse {test-name}, with each test separated by a space, to run one or more specific rehearsals
Comment: /pj-rehearse more to run up to 10 rehearsals
Comment: /pj-rehearse max to run up to 25 rehearsals
Comment: /pj-rehearse auto-ack to run up to 5 rehearsals, and add the rehearsals-ack label on success
Comment: /pj-rehearse abort to abort all active rehearsals
Comment: /pj-rehearse network-access-allowed to allow rehearsals of tests that have the restrict_network_access field set to false. This must be executed by an openshift org member who is not the PR author

Once you are satisfied with the results of the rehearsals, comment: /pj-rehearse ack to unblock merge. When the rehearsals-ack label is present on your PR, merge will no longer be blocked by rehearsals.
If you would like the rehearsals-ack label removed, comment: /pj-rehearse reject to re-block merging.

@mtulio
Copy link
Contributor Author

mtulio commented Jan 9, 2025

/pj-rehearse auto-ack

@openshift-ci-robot
Copy link
Contributor

@mtulio: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

Copy link
Contributor

openshift-ci bot commented Jan 10, 2025

@mtulio: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
ci/rehearse/periodic-ci-3scale-qe-3scale-deploy-main-3scale-amp-ocp4.14-lp-interop-3scale-amp-interop-aws d34d982 link unknown /pj-rehearse periodic-ci-3scale-qe-3scale-deploy-main-3scale-amp-ocp4.14-lp-interop-3scale-amp-interop-aws
ci/rehearse/periodic-ci-3scale-qe-3scale-deploy-main-3scale-amp-ocp4.13-lp-interop-3scale-amp-interop-aws d34d982 link unknown /pj-rehearse periodic-ci-3scale-qe-3scale-deploy-main-3scale-amp-ocp4.13-lp-interop-3scale-amp-interop-aws
ci/rehearse/periodic-ci-openshift-release-master-nightly-4.18-e2e-osd-ccs-gcp d34d982 link unknown /pj-rehearse periodic-ci-openshift-release-master-nightly-4.18-e2e-osd-ccs-gcp

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

Comment on lines +316 to +320
# temp workaround for
# https://issues.redhat.com/browse/OCPBUGS-45218
# https://issues.redhat.com/browse/OCPBUGS-46596
echo "ec2:DescribeInstanceTypeOfferings" >> ${PERMISION_LIST}

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like this has been restored after rebsae.

@mtulio
Copy link
Contributor Author

mtulio commented Jan 31, 2025

/test all

@openshift-bot
Copy link
Contributor

Issues in openshift/release go stale after 30d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle stale.
Stale issues rot after an additional 15d of inactivity and eventually close.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle stale

@openshift-ci openshift-ci bot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Mar 2, 2025
@openshift-merge-robot openshift-merge-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Mar 2, 2025
@openshift-merge-robot
Copy link
Contributor

PR needs rebase.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants