simulate: resource population

[joe-p]

Summary

When a user calls simulate with UnnamedResources enabled, simulate should suggest to the user how they can populate the resource arrays in their transactions to properly send the transaction group to the network.

Test Plan

• Test ResourcePopulator works with simple local (not group sharing) resources
• Test ResourcePopulator with group sharing
• Test ResourcePopulator resource limit detection with group sharing (ie. it is able to find the correct transaction to put a resource in)
• Test Simulate with ResourcePopulator functionality
• Test /simulate endpoint with ResourcePopulator functionality
• Write smaller tests for better ledger/simulation/resources.go coverage

[codecov]

Codecov Report

Attention: Patch coverage is 88.72549% with 46 lines in your changes missing coverage. Please review.

Project coverage is 51.97%. Comparing base (8fce49c) to head (b7bda2e).
Report is 18 commits behind head on master.

Files with missing lines	Patch %	Lines
daemon/algod/api/server/v2/utils.go	0.00%	29 Missing ⚠️
ledger/simulation/resources.go	96.45%	8 Missing and 5 partials ⚠️
ledger/simulation/simulator.go	66.66%	2 Missing and 2 partials ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #6015      +/-   ##
==========================================
+ Coverage   51.79%   51.97%   +0.18%     
==========================================
  Files         644      644              
  Lines       86511    86919     +408     
==========================================
+ Hits        44805    45175     +370     
- Misses      38834    38870      +36     
- Partials     2872     2874       +2     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[joe-p]

I believe all comments have been addressed at this point and test coverage is near 100%. The only problem is I'm still occasionally getting database table is locked when running tests locally. So far it's only happened with TestPopulatorWithGlobalResources. I tried just running this test and disabling parallel testing but I'm still seeing the same error occasionally. I believe this is just a problem with the test harness so not sure if it should be considered a blocker or not. I'd be interested to know if others can replicate.

[joe-p]

As I was working on SDK support and testing for full coverage of the API I realized I didn't have a test with full coverage of the API here. I found a bug that seems to be related to extra-resource-arrays. I will write a test for proper coverage of the API and then mark as ready for review once ready

Fixed in 791df53. See body for details

[joe-p]

The failing CI was from the non-deterministic behavior of the population algorithm. Turns out the table is locked message was correlated but not causal.

From the body of fd2c8dc:

The initial reason for desiring this was because of the non-deterministic behavior of maps made testing difficult. When thinking about it more, I realized that having deterministic population will also improve the developer experience since the same txn group will always get back the same populated resources. This change, however, does expose a potential problem inherit to resource population: order matters. As seen in the modified test, the determinstic order results in an extra resource in the extra resource array. In the future steps could be taken to try to improve the efficiency, but unless we go over every permutation of resource ordering there will always be cases where algorithmic population will not result in the most efficient resource packing.

[algorandskiy]

Relevant issue #5616

[algorandskiy]

LGTM, some minor suggestions/questions.

[algorandskiy]

@joe-p could you fix the reviewdog and respond to the review?

[joe-p]

CI is failing due to an unrelated test: TestP2PEnableGossipService_BothDisable

Re-triggered test run passing