Fix invocation overhead experiment #240
Conversation
WalkthroughThe changes update function and method signatures for two parts of the code. In the benchmark module, the function Changes
Sequence Diagram(s)sequenceDiagram
participant Caller
participant generate_input
Caller->>generate_input: Call with (data_dir, size, benchmarks_bucket, input_buckets, output_buckets, upload_func)
generate_input-->>Caller: Return {'output-bucket': output_buckets[0]}
sequenceDiagram
participant IO as InvocationOverhead
participant DC as DeploymentClient
participant CSV as CSVReader
participant Server as ServerResponse
%% run method sequence
IO->>DC: enforce_cold_start(..., _benchmark)
DC-->>IO: Cold start result
%% process method sequence
IO->>CSV: Read CSV file
Note right of IO: Skip header row
CSV-->>IO: CSV rows
IO->>IO: Process rows with extend_time_interval
IO->>Server: Retrieve timestamp from res.output["result"]["output"]["timestamp"]
Poem
✨ Finishing Touches
Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media? 🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. CodeRabbit Commands (Invoked using PR comments)
Other keywords and placeholders
CodeRabbit Configuration File (
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🔭 Outside diff range comments (1)
sebs/experiments/invocation_overhead.py (1)
165-171: 💡 Verification agent🧩 Analysis chain
Added extend_time_interval parameter to process method.
The
processmethod signature has been updated to include anextend_time_intervalparameter of typeint. This parameter appears to be required by the method's implementation, but it's not actually used within the method body.Do you need to actually use this parameter within the method implementation, or should it be marked as optional with a default value?
🏁 Script executed:
#!/bin/bash # Check all calls to the 'process' method to see if they provide the extend_time_interval parameter rg "\.process\(" --type py -A 3 -B 1Length of output: 466
Inconsistent Handling of
extend_time_intervalParameterThe
processmethod insebs/experiments/invocation_overhead.pynow includes anextend_time_interval: intparameter that isn’t actively used in its implementation. However, our search shows conflicting call sites:
- In sebs.py, the method is invoked with the
extend_time_intervalargument.- In scripts/process_experiments.py, the method is called without providing this parameter.
Consider either integrating the parameter into the method’s logic if it’s needed or making it optional (e.g.,
extend_time_interval: int = 0) to accommodate the call in scripts/process_experiments.py.
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (2)
benchmarks/000.microbenchmarks/030.clock-synchronization/input.py(1 hunks)sebs/experiments/invocation_overhead.py(4 hunks)
🔇 Additional comments (3)
benchmarks/000.microbenchmarks/030.clock-synchronization/input.py (1)
6-6:❓ Verification inconclusive
Function signature updated, but parameter appears unused.
The function has been updated to include a new
benchmarks_bucketparameter, but this parameter isn't used within the function body. This change aligns with the PR objective to fix the invocation overhead experiment by ensuring proper argument handling.
🏁 Script executed:
#!/bin/bash # Verify how the generate_input function is called in the codebase rg "generate_input\(" --type py -A 3 -B 1Length of output: 5990
Unused Parameter in Function Signature: Review
benchmarks_bucketUsageIn file
benchmarks/000.microbenchmarks/030.clock-synchronization/input.py, the functiondef generate_input(data_dir, size, benchmarks_bucket, input_buckets, output_buckets, upload_func): return {'output-bucket': output_buckets[0]}includes the new
benchmarks_bucketparameter per the PR’s intent. However, this parameter isn’t used within the function body. Notably, while some other benchmark modules (e.g., inbenchmarks/100.webapps/120.uploader/input.py) use this parameter, others (such as in this clock-synchronization benchmark) do not.Please verify whether:
benchmarks_bucketshould be integrated into the function’s logic (for example, to configure bucket operations), or- it should be removed (or at least annotated with a comment indicating its future use) to avoid potential confusion.
sebs/experiments/invocation_overhead.py (2)
226-226: Added CSV header row skipping.Good addition. This line correctly skips the header row of the CSV file, preventing it from being processed as actual data. This fixes a potential bug where the header row might have been incorrectly treated as a data row.
148-148:✅ Verification successful
Added benchmark parameter to enforce_cold_start call.
The
enforce_cold_startmethod now receives the benchmark parameter, fixing one of the argument mismatches mentioned in the PR objective. This change ensures proper communication with the deployment client when enforcing cold starts.
🏁 Script executed:
#!/bin/bash # Check the enforce_cold_start method signature to confirm it expects a benchmark parameter rg "def enforce_cold_start" --type py -A 2Length of output: 1186
Benchmark parameter addition in enforce_cold_start call is correct
After verifying the method signatures in multiple deployment client implementations (sebs/faas/system.py, sebs/aws/aws.py, sebs/azure/azure.py, etc.), it's confirmed that theenforce_cold_startmethod indeed expects a second parameter of typeBenchmark(namedcode_package). Passingself._benchmarknow properly aligns with the function signature, resolving the previous argument mismatch and ensuring correct communication with the deployment client.
- File: sebs/experiments/invocation_overhead.py, Line: 148
- Change: Updated the call to
self._deployment_client.enforce_cold_start([self._function], self._benchmark)
| @@ -276,7 +278,7 @@ def receive_datagrams(self, input_benchmark: dict, repetitions: int, port: int, | |||
|
|
|||
| # Save results even in case of failure - it might have happened in n-th iteration | |||
| res = fut.result() | |||
| server_timestamp = res.output["result"]["result"]["timestamp"] | |||
| server_timestamp = res.output["result"]["output"]["timestamp"] | |||
There was a problem hiding this comment.
Fixed timestamp access path in result object.
The access path for retrieving the server timestamp has been corrected from res.output["result"]["result"]["timestamp"] to res.output["result"]["output"]["timestamp"]. This change addresses one of the bugs mentioned in the PR objective regarding function calls with incorrect parameters.
The corrected path ensures that the timestamp is retrieved from the proper location in the response object, which is essential for accurate invocation overhead measurements.
|
@qdelamea-aneo Thank you for fixing the issue, much appreciated! We made multiple changes recently and didn't notice that it broke the experiments. Were you able to rerun the experiments successfully and fully, i.e., with the initial datagram exchange to establish network latency? |
The experiment to measure the invocation overhead doesn't work due to several bugs in
the code, in particular related to function calls with too many or too few arguments.
The changes proposed here make it possible to run the experiment successfully.
Summary by CodeRabbit