[Feature] Smarter spec files sharding / orchestration

[agoldis]

👋🏻

Let us know what functionality you'd like to see in Playwright and what your use case is.

Given a considerably large suite of tests, the aspect of efficient and consistent parallelization / sharding becomes important. In CI environment with multiple machines, an external service can provide an optimally sorted list of spec files for each CI agent - based on historical data, for example.

The proposal /request is to "democratize" the way test runner picks spec files, ideally incorporating some kind of plugins architecture, where one can implement different solutions for defining test suites selection and explicitly request the batch of next spec files to run.

I imagine that the agent-level implementation of parallelization can stay intact - i.e. each individual runner can keep its current behaviour.

Do you think others might benefit from this as well?

Yes, I think the whole ecosystem would benefit from this feature, especially organizations with larger tests suites. As a disclaimer, my particular interest is caused based on involvement with sorry-cypress community. Organizations / communities would be able to implement custom plugins that use historical data to improve the overall duration, easily adopt playwright for larger scale projects.

[aslushnikov]

Any orchestrations are already possible in the user land.

Say, for example, you have a set of spec files:

$ ls -1 e2e/
a.spec.ts
b.spec.ts
...
z.spec.ts

you can create a smart shard like this:

// shard.spec.ts

// get a list of specs for the shard - fetch from server or any other way; read config from process.env...
const filenames = ['./a.spec.ts', 'd.spec.ts'];
for (const filename of filenames)
  require(filename);

This already looks pretty flexible to setup any kind of smart configuration!

[agoldis]

Thanks for your prompt response @aslushnikov!

That's quite a flexible and creative approach - never realized it was possible! I've had a chance to play more with the concept and here are some questions / notes.

1. The suggested approach doesn't cope well when async function are involved

For example:

function* getFilenames() {
  yield './a.spec.ts';
  yield './b.spec.ts';
}

async function main() {
  for await (const filename of getFilenames()) {
    require(filename);
  }
}

main().catch(console.error);

That's because when require is getting invoked, transformer is already uninstalled from require extensions.

There are several workarounds I thought of:

• pre-transpile test file in advance. That's not optimal, using native playwright transformer tools would be more convenient.
• use a method other than require, e.g. Loader.loadTestFile (or some other 3rd party inline transpiler) - that's tricky because Loader isn't a public API

Ideally, one could just require (or preferably ESM import()) naturally... May be #7121 has something to do with this problem?

2. How to get access to CLI filters / spec files list?

The suggested approach assumes invoking the next CLI command, if I got it right:

npx playwright test ./tests/shard.spec.ts

It's still tricky to explore all the spec files - i.e. they should be either explicitly listed or shard.spec.ts would need some hint to discover the files.

Also, how one can combine test filters with the suggested approach?

---

I am imagining some kind of hook / plugins system that provides an execution context with:

• CLI arguments
• list of discovered spec files

and "feeds" the list of spec file to test runner, based on response from a remote service (any async operation, basically).

P.S.

Some approaches assume known # of machines, however often the amount of machines isn't deterministic - e.g. in CI environment that runs 100s of containers it's common to have several failed containers that never get to running actual tests.

[aslushnikov]

The suggested approach doesn't cope well when async function are involved

Yes, I'd suggest have a separate node script to generate shard and then run playwright test against that shard.

How to get access to CLI filters / spec files list?

The following should give you a list of all the tests:

npx playwright test --list --reporter=json