Test count and smart sharding count dont match #986
Comments
|
A shard can have multiple tests. The only time shard count matches test count is if every test is in exactly one shard. |
|
if the printing for smart sharding is for total number of test that will be put in shard, I expect them to be same. In original post, all numbers are for test count not for shard bucket count.
At the end of flank run, results shows 349 tests for app, and 1, 1, 2 and 16 for additional apks. Shouldn't smart flank use 349 for calculation of shard bucket? |
|
And what is actually bizarre is we should have 542 tests 😄 |
|
interesting, yeah there's probably a bug somewhere 🤔 I'll let the team take a deeper look! |
|
Here is obfuscated android_shards.json and flank.yml |
tl;drFlank does not support parameterized tests sharding. Every class with parameterized is considered as one test during shard calculation. Flank is using DEX parser to decompile apks and gather info about all the tests inside. As for now, Flank is unable to determine how many times a test in parameterized class is invoked. Due to this fact scans apks for any class with an annotation that contains @RunWith(JUnitParamsRunner::class)
...
@RunWith(Parameterized::class)and consider this class as a single 'test`, which means - 'put all test from this class into this shard' Example:@RunWith(JUnitRunner::class)
class A {
fun test1() {...}
fun test2() {...}
fun test3() {...}
}
@RunWith(Parameterized::class)
class B(param1, param2, ...) {
fun test1() {...}
companion object {
@JvmStatic
@Parameterized.Parameters
fun data() = listOf(
arrayOf(...),
arrayOf(...),
arrayOf(...)
)
}
}
@RunWith(JUnitRunner::class)
class C {
fun test1() {...}
fun test2() {...}
fun test3() {...}
}
@RunWith(JUnitParamsRunner::class)
class D {
@Parameters(value = [a,b,c,d]
fun test1() {...}
}For apk with classes like above Flank will have a list of tests* to shard:
So it gives 3 + 1 + 3 + 1 = 8 and let's say we want to have 2 shards so flank will calculate shards like
and will print (same numbers are used to print cache hit) To get outcome Flank uses FTL API which provides results for all the tests, regardless if it's parameterized or not. So flank creates outcome for all the tests ( Conclusions:
@bootstraponline @jan-gogo @Sloox @piotradamczyk5 @adamfilipow92 |
I think we already researched this and determined that parameterized tests can only be run by class?
It might be more accurate to say classes per shard when detecting parameterized tests? If there are no parameterized tests then it's true to say we are doing per method sharding. If there are parameterized tests, then we do a hybrid of class sharding + per test sharding. |
|
So basically we have 2 cases:
I am wondering if it won't confuse users. Sometimes there are tests, sometimes classes, I've already received one feedback and proposal if we could implement it simpler: That would also require to add The above approach is definitely much easier to implement. It's hard for me to evaluate the value of I think it will be valuable to get some feedback from the community? Or maybe you have some input |
😂 let's not add this wording to Flank I suggest we consult with our lead UX designer. |
|
Flank knows exactly how many tests and classes are being sent to firebase test lab, maybe we could do something like this:
|
I am not UX ¯_(ツ)_/¯ that was first that came to my mind :) |
I like it |
|
In the time of smart flank calculation, if you know there are parameterized tests, they can be excluded in smark flank success. Instead of 6 normal 1 parameterized test |
|
A shard may contain both individual test methods and/or test classes. We should be able to make smart flank aware of classes and cache the times. |
|
How about the case when either class count or test count equals 0? Do we want to print it that way: or should we skip it? Or skip it only for the class count? |
👍 , I was wondering if w should add the default time for class. For a test default time is 120s. I assume that a parameterized class has at least two sets of parameters, therefore default time for class can be 2x test time = 240s |
default time for classes makes sense to me, as I expect this might be different compared to individual test methods.
I think if there's a 0 we should skip it.
|
|
We have a parameter
|
We probably want a |
Describe the bug
Test result shows app module has 349 tests but smart sharding show and calculate with 232.
To Reproduce
It happens with out project. i dont have a repo to share.
Expected behavior
Test result and smart sharding should have same count.
Details (please complete the following information):
Flank 20.08.1
In the repo, there are Ignored, suppressed and parameterized tests.
With no orchectrator
The text was updated successfully, but these errors were encountered: