feat: apply view_tag to reduce sync time

[thanhdanh]

Summary

One of the most common complaints around using Monero day-to-day is the time that it can take to sync up a wallet before being able to send Monero. It seems important to speed up scan times. I think we can easily reduce the sync timing ~30-40% with 1 additional byte per output; call it the 'view tag’.

The principle is simple (and has likely been suggested before), make a shared secret between the sender and receiver, hash it, then record the first 1 byte in the chain (the view tag). Recipients calculate the shared secret, hash it, and check if it matches any view tags in the tx. If it does match, then continue checking for owned outputs in the normal way. It will only match about 1/256 of the time, so scan time will be primarily a function of checking view tags.

I did many timing tests and saw the difference between before and after applying view_tag. The duration of sync has been reduced around by around 30 -> 40%.

Depending on adding the threadpool module in monero-core-custom

Screenshots

Here is my test when syncing +30k blocks before and after using view_tag

Before

After

[feri42]

How much of the speed up is due to threading and how much due to view tags. Just wondering why implementing view tags depends on adding threadpool support?

Did I see correctly that we always choose only one thread, so the above question doesn't make sense?

[thanhdanh]

How much of the speed up is due to threading and how much due to view tags. Just wondering why implementing view tags depends on adding threadpool support?

Did I see correctly that we always choose only one thread, so the above question doesn't make sense?

@feri42 , good question.
I did a performance check about syncing the latest 100k blocks with 2 scenarios:

1. apply viewTag with parallel batches (~20 mins)

2. apply viewTag without parallel batches. (~20 mins)

They are pretty much the same as each other. No significant benefit from batching process (parallel handling)

3. Without view-tag

The duration is ~29 mins

[feri42]

Another question about view tags. Are we able to fetch all transactions using view tags? It it seems to me like sent transactions to different addresses would all have different view tags and during a refresh or an initial scan we do not know all of those view tags. So there are scenarios where we do not fetch all transactions.

[thanhdanh]

Yes, we must fetch all transactions. We don't know exactly which view-tag belongs to the sender. We have to let the wallet loop for each output of a transaction and then do compare the output's view-tag with the derived view-tag from the sender's keys. This is exactly the way that Monero Wallet is doing.

The view-tag is just the checkpoint to quickly decide to skip some EC operators, and we could reduce ~30% time to sync.

[feri42]

Yes, we must fetch all transactions. We don't know exactly which view-tag belongs to the sender. We have to let the wallet loop for each output of a transaction and then do compare the output's view-tag with the derived view-tag from the sender's keys. This is exactly the way that Monero Wallet is doing.

The view-tag is just the checkpoint to quickly decide to skip some EC operators, and we could reduce ~30% time to sync.

Ahh. I see. We look at every transaction. From the transaction we know the sender (?) and once we have the sender we can get the view tag for this sender (sender/recipient). And once we have the view tags we can skip non-matching outputs. Is that it?

[feri42]

Yes, we must fetch all transactions. We don't know exactly which view-tag belongs to the sender. We have to let the wallet loop for each output of a transaction and then do compare the output's view-tag with the derived view-tag from the sender's keys. This is exactly the way that Monero Wallet is doing.

The view-tag is just the checkpoint to quickly decide to skip some EC operators, and we could reduce ~30% time to sync.

Ahh. I see. We look at every transaction. From the transaction we know the sender (?) and once we have the sender we can get the view tag for this sender (sender/recipient). And once we have the view tags we can skip non-matching outputs. Is that it?

[thanhdanh]

Yes @feri42 , it's correct.

[feri42]

utACK