Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add tests to check response time in API #2204

Open
Tracked by #765 ...
seluianova opened this issue Apr 5, 2023 · 3 comments
Open
Tracked by #765 ...

Add tests to check response time in API #2204

seluianova opened this issue Apr 5, 2023 · 3 comments
Labels
feature New feature or request prio: low Not urgent, can be resolved in the distant future. question Further information is requested
Milestone

Comments

@seluianova
Copy link
Contributor

seluianova commented Apr 5, 2023

Motivation

In addition to #2091, we could also measure the API response time.

We probably don't want to generate a load, so we could just capture the current timestamp before sending the request and after getting the response, as an simplest option.

Questions to think about:

  • The response time may depend on the system, so perhaps we want to introduce a separate parameter to enable/disable this measurement?
  • Perhaps it could even be a separate set of tests and a separate CI job that does not cause the build to fail, but adds an informational comment like Code Climate?
@seluianova seluianova added feature New feature or request question Further information is requested prio: low Not urgent, can be resolved in the distant future. labels Apr 5, 2023
@seluianova seluianova added this to the Backlog milestone Apr 5, 2023
@seluianova seluianova mentioned this issue Apr 5, 2023
37 tasks
@timobrembeck
Copy link
Member

To gain at least a little bit of determinism, I think the CircleCI servers should be used as baseline for execution times. Maybe a first step would be to gather some statistics, e.g. for 20 CircleCI runs, what are the min/max/avg execution times for the API endpoints?
I still think there are probably edge cases where the timing will be completely off, e.g. because some task is scheduled later etc...

However, I think this really has a low prio because we don't know if this really adds much benefit in addition to the SQL query counter, whereas the other tickets in #765 are immediately very useful.

@seluianova
Copy link
Contributor Author

seluianova commented Apr 19, 2023

The response time may depend on the system

But as an option, we could use these tests not to check performance against some golden numbers, but against the master branch.
So we could potentially catch performance degradation.

@timobrembeck
Copy link
Member

But as an option, we could use these tests not to check performance against some golden numbers, but against the master branch. So we could potentially catch performance degradation.

Sounds like an interesting idea, could be worth investigating.
I'm not completely aware of CircleCI's internal load balancing, it could also be the case that they assign resources to containers quite dynamically, which could make the whole thing less predictable and could cause some test cases to have different execution time even if nothing else changed.

I think a good first step would be to write some code to measure the execution times and then let these tests run a few times with the same code base to see whether the results are somewhat consistent or whether there are high variances.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature New feature or request prio: low Not urgent, can be resolved in the distant future. question Further information is requested
Projects
None yet
Development

No branches or pull requests

2 participants