Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Run Percy less frequently #24731

Closed
Johennes opened this issue Mar 3, 2023 · 4 comments
Closed

Run Percy less frequently #24731

Johennes opened this issue Mar 3, 2023 · 4 comments
Assignees
Labels
A-Testing Testing, code coverage, etc. T-Task Tasks for the team like planning

Comments

@Johennes
Copy link
Contributor

Johennes commented Mar 3, 2023

We currently exceed our Percy quota about halfway through a monthly cycle. This is suboptimal because we're getting no value out of this tool half of the time.

To mitigate this, let's reduce the frequency at which we run Percy. Currently we have ~150 monthly commits or ~5 a day. I propose that we start out with 2 runs a day and see how far this gets us.

The current cycle ends on the 20th of March. So ideally we'd have this in place by that time.

@Johennes Johennes added T-Task Tasks for the team like planning Z-Infra A-Testing Testing, code coverage, etc. labels Mar 3, 2023
@Johennes Johennes changed the title Run Cypress less frequently Run Percy less frequently Mar 3, 2023
@luixxiul
Copy link

luixxiul commented Mar 6, 2023

I'm sorry if I get it wrong, but IMHO setting up a bot to let it run a Percy test only with a command issued on PRs labelled with X-Needs-Percy probably might work here without slowing down development which needs Percy tests, since it is said that using Percy is preferred to checking CSS styles with .should("have.css", ...).

matrix-org/matrix-react-sdk#10232 (comment), https://github.com/matrix-org/matrix-react-sdk/pull/9065/files#r923415568

This check should be captured by a percy snapshot, not explicitly checking the style. These e2e tests should be testing from the POV of the user to see that 1) things work as expected and 2) look as expected.

@Johennes
Copy link
Contributor Author

Johennes commented Mar 7, 2023

We actually have that logic of running Percy only on PRs that have the X-Needs-Percy label already. This issue is about reducing the frequency at which we run Percy on develop because it's making us exceed our quota half-way through a monthly cycle already.

@luixxiul
Copy link

luixxiul commented Mar 7, 2023

Right, I did not notice that Percy ran on develop branch too, which in fact is documented here: https://github.com/matrix-org/matrix-react-sdk/blob/develop/docs/cypress.md#percy-visual-testing

@t3chguy
Copy link
Member

t3chguy commented Apr 25, 2023

We did this

@t3chguy t3chguy closed this as completed Apr 25, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-Testing Testing, code coverage, etc. T-Task Tasks for the team like planning
Projects
None yet
Development

No branches or pull requests

4 participants