loadgen "vault" task: zoe is leaking BLD Payment objects?

[warner]

I'm running perf tests using https://github.com/Agoric/testnet-load-generator on a branch in which #3482 is fixed, which means we have meaningful GC taking place. The "tap-faucet" and "trade-amm" tasks appear to have flatten out their object import/export graphs (i.e. their vats are dropping everything they import at the end of each cycle). However the "create-vault" task is not. I'm seeing what appears to be a leftover BLD Payment at the end of each round, exported by vat-mints and imported by vat-zoe.

Zoe seems to send a .withdraw(0 BLD) message to vat-mints, and it's the resulting Payment that is not being dropped by vat-zoe.

@katelynsills if you get a chance, maybe we could walk through the log and try to understand what's going on. The create-vault source code is here:

• https://github.com/Agoric/testnet-load-generator/blob/main/loadgen/task-create-vault.js
• https://github.com/Agoric/testnet-load-generator/blob/main/loadgen/agent-create-vault.js

The task file installs the agent file into the solo node, then once per minute it calls vaultCycle, which instructs the agent to doVaultCycle. The agent is intended to open a vault and then close it again right away. The output of the loadgen cycle shows non-zero balances for both RUN and BLD (the BLD balance remains stable at 6.5, the RUN balance drops by a small amount each time, as expected with fees).

The withdraw of 0 is puzzling, and I suspect is related to whatever is leaking.

[katelynsills]

Thanks for finding this. I'll be investigating on my end. Since it involves packages/treasury as well as Zoe, let's involve @Chris-Hibbert too.

[warner]

We weren't able to figure out what exactly was leaking, but we determined that the treasury code is using some API in a weird way (and has a bug anyways). So we decided to first fix that issue, then re-evaluate to see if it's still leaking the object.

[katelynsills]

See #3495 and #3497 for the bug in the Treasury and the unusual pattern of returning payments outside the usual payout path. We need to fix the bug before the next testnet phase.

[Tartuffo]

@warner Do we think the treasury code API change fixed this problem?

[warner]

Good question, @mhofman do our daily loadgen runs give us enough information to see whether the Vault task still has a problem or not? I know @arirubinstein 's testnet load is using AMM trades, and is seeing a leak of some sort, so either the AMM contracts or his load generator still have a problem. But it'd be great to know whether the Vault code/loadgen still has a similar problem.

[mhofman]

Now that #5653 landed, once the loadgen has run on that revision, I or @warner will check if the number of kernel objects and kernel promises is stable across stages.

[warner]

Our quick look at a few hours of data suggests the AMM+vault loadgen is not growing the number of kernel objects significantly.

[dckc]

What else is there to do on this? Or did you mean to close it, @warner ?

The Vaults tag made it show up as an Inter protocol agenda candidate; I think it fits SwingSet better.

[mhofman]

If there was a problem it could have been either, but since we're not seeing any accumulation, I'd be in favor of closing.