Debug_traceBlockByHash returning "insufficient funds" error on block 19888327

[Enigmatic331]

System information

Geth version: geth.exe version 1.14.3-stable-ab48ba42
CL client & version: beacon-chain version Prysm/v5.0.3/38f208d70dc95b12c08403f5c72009aaa10dfe2f. Built at: 2024-04-04 18:48:31+00:00
OS & Version: Windows

Expected behaviour

Debug_traceBlockByHash should return the trace dump for block 19888327

Actual behaviour

A tracing failed message was returned instead

C:\Users\Administrator>curl http://localhost:8545 -m 300 -X POST --header "Content-type: application/json" --data "{\"jsonrpc\":\"2.0\", \"method\":\"debug_traceBlockByHash\", \"params\":[\"0xcc1d35533ea26392bfa91944b3f6d1f446581b05f5f4b35dea93d558b08d353c\", {\"tracer\":\"callTracer\"}], \"id\":1}"
{"jsonrpc":"2.0","id":1,"error":{"code":-32000,"message":"tracing failed: insufficient funds for gas * price + value: address 0xc79d40C897FbD912a7cD29E318a0D41F579793f6 have 2469004274997060 want 3963014826093176"}}

If I independently trace the two "out" transactions on this block for the address 0xc79d40C897FbD912a7cD29E318a0D41F579793f6 , it seems to return results correctly.

C:\Users\Administrator>curl http://localhost:8545 -m 300 -X POST --header "Content-type: application/json" --data "{\"jsonrpc\":\"2.0\", \"method\":\"debug_traceTransaction\", \"params\":[\"0xefa4345e670ef386c0c1e076625c1a7627df192adf5ed8b7f5a8ec27c14ccdfd\", {\"tracer\":\"callTracer\"}], \"id\":1}"
{"jsonrpc":"2.0","id":1,"result":{"from":"0xc79d40c897fbd912a7cd29e318a0d41f579793f6","gas":"0x927c0","gasUsed":"0x60126","to":"0x5f6ae08b8aeb7078cf2f96afb089d7c9f51da47d","input":"0xba1f77e80000000000000000000000000000000000000000000000000000000000000000","calls":[{... etc

C:\Users\Administrator>curl http://localhost:8545 -m 300 -X POST --header "Content-type: application/json" --data "{\"jsonrpc\":\"2.0\", \"method\":\"debug_traceTransaction\", \"params\":[\"0xb3552b1f6e4c97349ca45ae34bb831c04c571fd4fa2110481d4aae851e8a0331\", {\"tracer\":\"callTracer\"}], \"id\":1}"
{"jsonrpc":"2.0","id":1,"result":{"from":"0xc79d40c897fbd912a7cd29e318a0d41f579793f6","gas":"0xa410","gasUsed":"0x5208","to":"0x742c5ce550ec2e1a78b338904978ec9004eb065f","input":"0x","value":"0xc8ed706a30328","type":"CALL"}}

Seems to be affecting more than 1 of our nodes, potentially other blocks as well as we received some reports about indexed traces showing a transaction failing, however when the tx is traced individually with debug_traceTransaction, no execution reverted seemed to be found.

[holiman]

That block contains three transactions concerning that account

• OUT 0xb3552b1f6e4c97349ca45ae34bb831c04c571fd4fa2110481d4aae851e8a0331
• OUT 0xefa4345e670ef386c0c1e076625c1a7627df192adf5ed8b7f5a8ec27c14ccdfd
• IN 0xd5fd51583eab8fdbb885b017683ee9e5d6b2154103261c49a996cfc542b3dc23

[Enigmatic331]

The last tx could also run through debug_traceTransaction, though I didn't raise it as it didn't seem to concern 0xc79 as a sender 🙈 - Assuming that the insufficient funds for gas * price + value: address 0xc79d40C897FbD912a7cD29E318a0D41F579793f6 have 2469004274997060 want 3963014826093176 error targets address 0xc79 sending a transaction to a contract/EOA.

C:\Users\Administrator>curl http://localhost:8545 -m 300 -X POST --header "Content-type: application/json" --data "{\"jsonrpc\":\"2.0\", \"method\":\"debug_traceTransaction\", \"params\":[\"0xd5fd51583eab8fdbb885b017683ee9e5d6b2154103261c49a996cfc542b3dc23\", {\"tracer\":\"callTracer\"}], \"id\":1}"
{"jsonrpc":"2.0","id":1,"result":{"from":"0x742c5ce550ec2e1a78b338904978ec9004eb065f","gas":"0xa410","gasUsed":"0x5208","to":"0xc79d40c897fbd912a7cd29e318a0d41f579793f6","input":"0x","value":"0x15bc4cf4757ac0","type":"CALL"}}

[holiman]

So, looking at the IN one state changes:

Before that, the account has nothing. So it's not just a case of "the block-tracing doesn't commit the transaction state changes and executes every trace on the block prestate", because then the error would say have 0 want...

[Enigmatic331]

Ahhh you're right... Now where did the number 2469004274997060 came from...

[mtbitcoin]

Also block# 19888387 , #19888680 with similar issues if you need extra samples

[s1na]

Are you using the hash scheme or path scheme? If you first synced your node before v1.14.0 and didn't explicitly specify --state.scheme=path then it is hash scheme. v1.14.0 flips the default to path but only if you do a fresh sync.

[Enigmatic331]

We're using hash for our nodes (especially for this one where we are running archive nodes) 🙏

[quickchase]

Yeah, archive nodes have to be hash, path isn't supported, but this happens on either, I believe

[holiman]

I've been investigating this a bit, haven't found any smoking gun. The callTracer is a 'native' tracer, so it doesn't go into the more complicated parallel prestate-execution path, but only the simpler step by step tracing of transactions.

I have found one thing that looks quite odd, https://github.com/ethereum/go-ethereum/blob/master/eth/tracers/api.go#L985 , where we use a vmconfig with no basefee checking vm.Config{Tracer: tracer.Hooks, NoBaseFee: true}, and a mostly empty txcontext with e.g. zero gasprice vm.TxContext{GasPrice: big.NewInt(0)}. I don't see how that could be correct.

[holiman]

It seems, that earlier, it generated the txContext from the msg, but it was changed to the 'empty' form here:

064f37d#diff-bd288cd51c1874c7e65ce0d35047c0a98366479c8924606c9efc577f6a98f75cL947

@s1na any thoughts why?

EDIT: Nevermind, it seems that it's now set within ApplyTransactionWithEVM

	// Create a new context to be used in the EVM environment.
	txContext := NewEVMTxContext(msg)
	evm.Reset(txContext, statedb)

[celochoi]

I checked there was no problem with 1.13.15.
Does this issue only occur in 1.14.x?

[holiman]

@s1na the setting of NoBaseFee: true in combination with vm.TxContext{GasPrice: big.NewInt(0)}, will have consequences. Later on, during vm.NewEVM:

	if config.NoBaseFee {
		if txCtx.GasPrice.BitLen() == 0 {
			blockCtx.BaseFee = new(big.Int)
		}
		if txCtx.BlobFeeCap != nil && txCtx.BlobFeeCap.BitLen() == 0 {
			blockCtx.BlobBaseFee = new(big.Int)
		}
	}

That is: we will set the block ctx fields to zero. Thus overwriting the fields that we constructed earlier, in eth/tracers/api.go:

		blockCtx  = core.NewEVMBlockContext(block.Header(), api.chainContext(ctx), nil)

This will affect the opcodes BASEFEE and BLOBBASEFEE.
If anyone can provide me with the full opcode by opcode output of https://etherscan.io/vmtrace?txhash=0xefa4345e670ef386c0c1e076625c1a7627df192adf5ed8b7f5a8ec27c14ccdfd, that'd be greatly appreciated. I'd like to see what opcodes are executed.

[pengin7384]

@holiman
Did you mean debug_traceTransaction response?

Request

{
    "method": "debug_traceTransaction",
    "params": [
       "0xefa4345e670ef386c0c1e076625c1a7627df192adf5ed8b7f5a8ec27c14ccdfd"
    ],
    "id": 1,
    "jsonrpc": "2.0"
}

Response (w/ geth 1.13.15)
response.json

[quickchase]

Was the above generated with 1.13 or 1.14? I have 1.13 archive nodes and I have Erigon, Nethermind, and Besu archive nodes, if you want any of those, lemme know 😄

[holiman]

Yep,

It does invoke BASEFEE

The value is 0x15818a10a, and later on does some LT checks and jumps depending on it's value

This is most likely the culprit

[pengin7384]

@quickchase It was generated with geth 1.13.15.

[Mazzika1]

Okay

[s1na]

This should have been fixed via #29811. I would appreciate if someone can use latest master and confirm they don't see the issue anymore.

[mtbitcoin]

@s1na this looks fixed now, tested the the unstable nightly build from 28th May. It solves the issue above and another related issue where the transaction trace was showing an error (where it should not have been)

[s1na]

Great, closing then. Release coming up next week!